Prairie State Update: Results Driven Accountability in Illinois

00:00:21.80 >> Hello. I'm Amy Bitterman, and this is IDC's A Date with Data. On this episode, I have with me Kristina Holloway, special education monitoring lead with the Illinois State Department of Education, and she's going to be describing Illinois' annual LEA determinations process. IDEA requires states to make determinations annually about the performance of each local agency, or LEA, and Illinois has made some changes to the process over the last few years, and really excited to hear about those, and what this process looks like in Illinois. So welcome, Kristina. Thank you so much for being on.

00:01:00.06 >> Thank you for having me.

00:01:01.68 >> Can we start out just saying a little bit about yourself, your role, how long you've been with the Illinois State Board of Education, background, that kind of thing?

00:01:10.44 >> Absolutely. I was a special educator prior to coming to the state Board of Education. I did spend 20 years in the classroom, both at the elementary and at the high school level. I made the leap to the state Board of Education in the summer of 2021, so I just recently celebrated my fourth year, anniversary at the state Board of Education since --

00:01:35.39 >> Happy anniversary!

00:01:36.63 >> -- thank you! Since coming to the state Board of Education, I actually wore many hats there. I started off as a principle consultant for specialized populations, so my work there really was focused on students with a specific learning disabilities, emotional disabilities, and autism. So I oversaw multiple technical assistance projects that we funded through our state Board of Education with IDEA discretionary dollars, and other state funds.

00:02:09.90 >> Then I moved into the state performance plan annual performance report coordinator, where I got a lot of background of the discussion we're gonna have regarding LEA determinations today. Learning about it, I dove a lot into the data, which I found I loved, and really wanted to figure out how that data related to supporting students with facilities in our state, and try to improve those outcomes for those students.

00:02:37.60 >> Then my next role was to go into my current role, which is as the special education monitoring lead. I feel like I kind of just stair-stepped up as I took my experience from the classroom to different levels, and kind of carried that with me all along.

00:02:51.65 >> Yeah. It's great having that long history in the classroom, and then bringing kind of that perspective to the state. I think that makes a big difference, probably.

00:03:00.92 >> Yeah, absolutely. It really does help reach the LEAs, I think, because I can speak on their level, and I understand where they're coming from when they do discuss some of the struggles they may face, or how to support, you know, not only their educators, but also the students that they serve as well.

00:03:17.74 >> Yeah. Absolutely. All right. So let's start out by talking about how you were making LEA determinations in the past, and why you decided to make some changes to your process.  

00:03:30.22 >> Sure! So some of the changes that took place prior to my arrival, they actually took place prior to 2020. And back before 2020, it used to be really focused on those compliance indicators, such as 9, 10, 11, 12 and 13. We included our single audit from [INAUDIBLE] and timely correction of noncompliance, of course, on those requirements. If a district didn't meet the compliance requirements for those indicators, they would receive a score of 1 automatically for that indicator. But if they did meet the requirements for that other state target, or the compliance requirement, they would receive a 4. So there wasn't really much separation between the scores. It was either automatic 1 or an automatic 4.

00:04:11.91 >> Yeah.

00:04:12.61 >> Single audit findings were an exception to those, though, because those were a sliding scale, and it was really related to the type of finding, and how long the finding may have been going on. Was it something that was systemic that it happened previously?

00:04:26.63 >> Mm-hmm.

00:04:27.92 >> Then LEAs would receive an average score of their determination, so their average score was also established with an 01 and a 4, as opposed to a percentage score. So that is kind of how they were previously done.

00:04:42.90 >> What made you all decide to make some changes?

00:04:46.98 >> So right around that time, I OSEP came out with their results-driven accountability system. We know it now as RDA. So [INAUDIBLE] decided to kind of follow suit with high end work with stakeholders to come up with our own results- driven accountability system to kind of align with our LEA determination. So during that time, during those stakeholder meetings, we looked at all the different results indicators as well, and the state targets, and kind of aligned it with the upcoming changes for the SPP-APR, you know, the new cycle with that, and put it all together. So through that feedback is, we did choose to move to more of a RDA approach, so focusing not just on that compliance, but also looking at the results pieces as well so that we could kind of tie that together, because as we know, compliance is great, but we also see how the students are progressing and what's going on to truly understand the real landscape of our state, and where those needs need to be addressed.

00:05:52.68 >> Yes. Absolutely. So that was about 2020?

00:05:56.44 >> Yes.  

00:05:57.14 >> Okay. All right. So once you had your stakeholders engaged and kind of doing your analyses to decide what you wanted the current process to look like, can you tell us what that does look like? What that process has been like since you made those changes and --

00:06:14.56 >> Sure. 

00:06:14.78 >> -- if there are any things in particular that have really worked well in this newer process?

00:06:19.68 >> Oh, sure. Yeah. So our current process now is, we do include results indicators, including Indicators 1, 5A and 6A, and our LEA determinations, of course, we continue to include the requirements from OSEP with the compliance indicators for the 9, 10, 11, 12 and 13.  

00:06:40.42 >> Mm-hmm.

00:06:40.93 >> We also have to incorporate that timely correction of noncompliance, our single audit findings, as well as that timely, reliable and valid data component. We do mesh those together, and we typically send our LEA determinations out each September. And what's nice about sending them out in September, and it is [INAUDIBLE] throughout the years, we have toyed around and kind of played around with those timelines, of when did those go out -- what's nice is that we start right at the beginning of the school year for everyone. So we're getting -- you know, we're working with the districts immediately so they can start implementing those things at the beginning of their school year, and not have to do necessarily things over the summer. If we would do it in May, they would have people in and out during the summer, and not really get to start that work at that time.

00:07:31.65 >> Yeah.

00:07:31.88 >> So we like being able to provide them that information right from the get-go, so they can get going on any corrected action, and then they can also start working on any kind of implement plans that they need to do to help with their data reliability, and help improving their systems, ultimately.

00:07:51.15 >> Uh-huh. Yeah, that makes a lot of sense, depending.

00:07:53.31 >> Yeah. And each score, we did also change our score range, so currently each score is based off of 0 through 3, based on their performance. One thing that we noticed with that is, we didn't want it to be an all or nothing, because we do understand that LEAs, you know, their data can be influxed, you know, based on the students who come in and out with mobility throughout the state. So we wanted to give some LEAs that opportunity to not have to lose all of their points, especially if after a year of maybe having a decline, they did show improvement, even if they weren't meeting the state target.

00:08:32.27 >> Uh-huh.

00:08:32.61 >> We wanted to recognize the improvement that they were showing.

00:08:36.10 >> Yeah.

00:08:36.20 >> So we did make that more of a sliding scale, with the exception of our timely correction and noncompliance component. So it does make it a little big nicer, and kind of also works, you know, for the LEAs so that they can show even their school wards and parents that, hey, we are making progress. I know we didn't get a 3, but it's okay. We went—you know, our percentage did go up. And they have that input, too, on what that looked like, so they have some ownership of what that process is, too.

00:09:10.64 >> Uh-huh.

00:09:11.42 >> So currently, LEAs who get an 80 percent or higher on their LEA determination, we consider those Meet Requirements. And unless they have any finding of noncompliance related to the compliance indicators, they have a multitude of universal supports that they can have access to, but there are no required corrective actions for those LEAs. LEAs who score 75 percent, but less than 80 percent, we consider those Needs Assistance. If they obviously did that for more than one year, then they would be Needs Assistance two more years, and they would have corrective action that they would need to complete throughout the year, aligned to what the regulations were to require them to do. They do need to utilize technical assistance, they need to come up with a corrective action plan for improvement. Of course if they have any findings or noncompliance related to the compliance indicators, they would have corrective action related to those as well.

00:10:07.09 >> And then any LEA that would score less than 75 percent we would consider Need Intervention. We recently just shifted to align more with the regulations, and a one-year need intervention, they only get a low 75 percent for one year. We let them have just universal support, of course if they have findings of noncompliance, as with the others related to those compliance indicators. Everyone would have corr--they would have corrective action like the others, as well.  

00:10:35.52 >> Uh-huh.

00:10:35.84 >> But once they hit that for two years, we have aligned that with our NHU process, so that they also have a warning system, so they align together. And then once they hit that three years, then we are perfectly aligned with what the regulations require, they are going to have more instance of supports, regular meetings with a principle consultant, technical assistance, require immense and corrective action requirements as well, to help them improve so that we can try to avoid that needs substantial intervention level.

00:11:07.20 >> Yeah.

00:11:07.66 >> We've also, with this shift, noticed that some may go from that NE2 level, which we consider tier 2, two needs intervention for one year. If that happens, or we see any decrease, we are going to still require the district to complete corrective action [INAUDIBLE] from that meeting, because they are not improving their score.  

00:11:26.89 >> Mm-hmm.

00:11:27.19 >> So we will be working with them to try to get that score up so that in the future, that we don't have to get to a significantly more intensive level, because to us, that's going to signify they need more support, so what else can we do for the, really, LEAs? So that is something that will happen this year, too.  

00:11:46.87 >> Are there particular areas that you feel like have worked really well in this process that you've been using now for a few years? You've heard feedback from districts in terms of how they feel about the process, that kind of thing?

00:12:00.70 >> Yeah. I think moving from that needs intervention one year, and having districts be required to do extremely intensive actions was helpful last year. I think we have such a diverse population in Illinois. We serve districts that are rather large. We have extremely large districts, especially in the northern part of our state down to the southern part of our state, and even in some of the Western/Eastern regions where we may only have an LEA that has 100 students in it.

00:12:31.38 >> Wow.

00:12:32.27 >> So utilizing the way we do our points can sometimes almost not be a disadvantage, but it can cause a smaller LEA to have less points available to them, and also more of an impact to their score because there will be certain things that do not impact them. You know, we obviously have LEAs that are only high school, we have LEAs that are only going to be elementary. So some of those indicators that we look at, even for compliance, don't apply to them. So by kind of giving them that break, and it's nice that that is allowable, allows them to kind of, oh wait -- look at their data and go over what we have to do to improve it. And they're starting with a smaller population already. So even on those results indicators, like 5A, 6A, their percentage might just naturally be less because they're working with a less number of students. So, you know, one or two students impacts them --

00:13:30.66 >> Right.

00:13:30.79 >> -- a little bit more than it does a large LEA.  

00:13:33.58 >> Yeah.

00:13:33.94 >> So I think that that really kind of gave them a little bit of a breath, so to speak, so they weren't, like, oh gosh, we've only done this for one year. This only happened once.

00:13:45.94 >> Right.

00:13:46.63 >> And looking at more systemic, not just a one-time, like okay, is this a systemic issue? Is this continuing? I think through our process, we just continue to analyze it year after year to see how do we improve it? What does the data tell us? Who is needing the most support? And then kind of building that, so ultimately we do identify the districts who do need the most support within our state, and then provide it to them.

00:14:15.42 >> Yeah. Yeah, are there any areas that you feel like you'd like to improve or make changes to?

00:14:22.18 >> So one of them -- and I forgot to mention this, I apologize -- one of the other things that we did add last year, and I think that this was also really great for LEAs, and we're getting ready to start this again, is that we have implemented a preview window for the LEAs so they are able to go into one of our data systems, we call it the "I-Star Plus system, it's where all the IEP information is held. It's not the actual IEPs themselves, but the student information, and regarding their EE codes, their placements, you know, their initial evaluations, transition plans, etcetera, we have in that system. So we do house an LEA determination window for them. And so they can go into that system about a week prior to the actual issuance of the LEA determination and look at their data ahead of time. And what's nice about that is, they can identify if something doesn't look right, they can bring it to our attention or they can kind of get a preview and go, okay, we know what's coming. So they're not maybe blindsided, per se. And we used to have districts that would try and do an appeal about their data, and they would think that we have some secret data that's not theirs, but it would be --  

00:15:34.19 >> That they don't, no doubt.  00:15:35.15 >> Yeah. It's not theirs that --, well, it is theirs. So it gives them that opportunity to kind of look at it. So with this process now, we have actually taken away the appeal process. We're giving them their data to preview ahead of time, to identify if they notice anything. And last year, we did get a lot of feedback, which was nice. It actually ended up being an error on our side. We recognized, oh, something didn't look right. So we were able then to troubleshoot on our side. And, you know, errors happen on both sides. So that was a really nice way to know that they were looking at it. They were intrigued by the information, and then they could report back to us. So hopefully, that doesn't happen this year. So far everything looks really, really good.

00:16:19.14 >> Good.

00:16:19.80 >> And we're about ready to open up that window, actually. I think we're scheduled in 10 days, and we're going to get that opened back up for them, so they'll be able to go in and see that data ahead of time.  

00:16:30.86 >> Yeah. That's really cool. I don't know that I've heard about other states doing that. That's a really interesting and unique opportunity to give districts.

00:16:41.01 >> Yeah. It's not like a pre-finding correction, or anything like that, that there's nothing allowed there.

00:16:46.93 >> Yeah.

00:16:47.02 >> It just more is, like, hey, let's do that. And we really based that off of how we provide our school district designations for ESSA. So they have a similar process that they use, and they can -- the districts can go in and the different schools can go in and see what that designation is going to be, and what their data looks like. So we really mirrored it off of a system that was existing within another department, our accountability department. So it's kind of nice to have a similar kind of system.  

00:17:16.03 >> Yep, the consistency, and especially with the districts, if they're also using ESSA, having the same kind of process in place that they're used to, that putting that in place makes a lot of sense.

00:17:27.46 >> Absolutely. Yeah.

00:17:29.07 >> Do you have an example you can talk about where you've used this process to really work with the district to help improve the quality of their data for students with disabilities?

00:17:40.65 >> Absolutely. I have several examples. One of the reasons we did want to adjust how we do the levels of support when it came to the determination was because when I first came into the role as SPP-APR coordinator, we did our LEA determinations. And I continually had phone calls from our smaller school districts, saying, hey, we only have six kids exiting, so for Indicator 1, their graduation, right, it was really impacting their graduation rate, because they would have students that were going into transition programs, were not going to earn a regular diploma --

00:18:17.86 >> Yeah.

00:18:18.23 >> -- and appropriately so. They were being appropriately placed and being provided the appropriate services through transition, which those students are allowed to receive. It was really, really tanking their Indicator 1 score. Then because they were so small, they may have only had less indicators than another district, you know, must because of the nature of the type of district it was.

00:18:42.46 >> Yeah.

00:18:42.56 >> So I kind of took a deep dive into that and heard it from multiple school districts in kind of the same thing, and thought, oh my gosh, we really do need to look at this, and we need to make sure that they understand how to get the data, what does that rate look like, and how we then pay attention to that when we align our systems as well, and the supports that are provided at each level of that LEA determination. So I took a deep dive with a lot of those districts, got into our systems. We analyzed that exit data, and really kind of looked at those students. Sometimes we caught errors with what was actually being coded, so that was really good. Then they were able to learn more about the systems of where that data could be accessed from, too, so that they could take it and kind of talk to their administrators in their building, and their educators in the building so that they all had a better understanding of, wait a second, you know, are our systems really aligned properly? Are we getting the correct data entered and submitted where it needs to go? So that was really a good piece of utilizing the data that comes from this, and kind of taking a deep dive.

00:19:55.50 >> Most recently though, we've been working with our larger school district in our state, and they have sought out needing a lot of support when it comes to their data. We have worked really, really diligently to build a great partnership with them, and to open that communication to truly show growth and improvement. And so we recently went on site actually with some IDC staff, and with their support, to really dive into the indicators as a whole, but then specifically those indicators that are included in our LEA determinations process. And it was fascinating because I think that sometimes there is a disconnect between the state, the districts, and then what goes on at the federal level, too.

00:20:42.86 >> Yeah.

00:20:42.95 >> What is required? What's coming down on the state? And what do we have to do as an SEA? And then what do we require the LEAs to do? You know, one of the presentations they did, of course, was Miguel's story, and the indicators don't go in order in Miguel's story. And they kept saying, "Well, why won't they be in that order? Why aren't they numbered in the way you go through Miguel's story, when you start off, if that indicator, 12 level or Indicator 11 with the child [INAUDIBLE], why doesn't it start where it starts?" So it was a really good conversations, and kind of helped open their eyes as to what is required from the SEA, and then why the LEAs, and then why the LEAs have to then submit that type of data. And it made -- utilizing Miguel's story makes it more real.  

00:21:31.17 >> Mm-hmm. Yeah. Yeah.

00:21:31.64 >> It puts it on a student, so they're able to see it through a student's eyes, and it is eye-opening for them. So that was really positive. That was in the spring that we did that. We went on site, and since then we have just continued to have those discussions of how we are going to look at that data, how we are going to train more of their staff regarding the data, the why, the where it comes from. Like are there issues within their own data system that need to be corrected? So we've had some really good conversations to really break it down, and to build better processes for the district itself, because being such a large district it can be very disconnected, and it is difficult when you're managing that many schools and you have all that data coming into a central place.

00:22:19.52 >> So what we'd like to do in the future, and then potentially use them as an example for other LEAs, is to have them go through that LEA data processes document to kind of build their data processes.  

00:22:34.41 >> Yeah.

00:22:34.52 >> We have done it at the state level. It was wonderful, because we were able to connect all the pieces together, and understand how everybody's role kind of fits into that. So we know that that will be really instrumental in working with our largest LEA on that, and then utilize that as an example for other LEAs to kind of learn from in the future, too.

00:22:57.74 >> Yeah. That's great. And just using kind of the determinations process you've put in place is like that springboard to really be able to dig deeper --

00:23:06.80 >> Yes.

00:23:06.93 >> -- and work very closely with your LEAs. Yeah, the data process toolkit from IDC is a great way to sit down with your districts and help them really document and talk through in excruciating detail all of their processes and procedures. So yeah, having kind of all those pieces come together, I think, really sounds like a great process.  

00:23:31.61 >> Yeah. And I think it's important as we recognize the state of education, not only in our state, but across the nation. And the turnover is real, and we can't ignore that. So having something in place for when that turnover happens, and building succession planning is just huge, because otherwise, when people leave, no one knows what happened. And so if you can document it, then at least somebody can pick it up and kind of learn from that. They may have a lot of questions, and it makes total sense to have that. But I just think we're really wanting to help our LEAs understand the importance of that, even in the busy day-to-day, that this is the foundation, and this is what's important because you have to use that data to really identify where are the greatest needs, and how are you going to support those students that you serve?

00:24:21.94 >> Yes. Absolutely. So other -- you mentioned trying to model the process you've been doing with one of your districts, in the future with other districts. Are there other plans you have or other things related to determinations you're thinking about doing in the future?

00:24:38.81 >> Well, we are getting ready to roll out [INAUDIBLE] a Power BI-4 to that I-Star Plus system that I mentioned before. That Power BI system has been quite an undertaking, honestly. And I'm really excited about finally getting that rolled out for the LEAs to be able to access their data, analyze their data. It's not just going to be the indicators that we utilize for LEA determinations. It will actually include all of them, 1 through 14 for them. And it gives them trend data as well over a three-year span so they can look at it. There's visuals with it, and really see what's going on with that data from year to year. So I'm really excited that they're going to have that. I can't wait for that to be put in place. We are in the final states of that, so that's exciting.

00:25:29.92 >> Great.

00:25:30.10 >> And I think that'll spurn a lot of conversation, so I think when they see that, we'll probably get districts reaching out and want to know more, and want to dive in of where it came from. But it gives them -- it just puts in data at their fingertips. It is a little hard, I would say, currently for them to pull it all together, so having it all in one place is going to make it a lot nicer for them. They can just go in one spot and just look, and kind of analyze it from there. So as we build that, too, we may end up looking at the supports that we do at the tiered level, you know, Tier 2, Tier 3, you know -- do we implement the LEA data processes document as a requirement at one of those levels of support, too, just because if they're having continual issues, it's not just a one-off.

00:26:19.82 >> Maybe they need to look at that and really build that in. So that would be a partnership that we could build with the LEAs too, and really look at their data and figure out, where are the questions? Where's the support needed? That utilizing our largest district potentially as a partner with them as well, where they could ask them questions, too, about how they did it. So that's kind of been a thought of moving forward, about what we want to do. And we'd look at the data every year. We'd make sure that everybody is completing things for their data. And if they're not, reaching out to them, but just trying to really hone in on those identified needs, and also utilizing the data to drive our technical assistance projects. What do we need? What are we missing? So if the data is saying something and the districts are telling us as we're working with them through specific corrective action processes, or other improvement processes, if we identify a need, too, then we can look into adding that technical assistance support, too. And we're, like, this may need to be something we have as a state level, too, for support. Yeah.  

00:27:27.47 >> Yeah. I mean, it seems like that Power BI, those reports, not just for the individual districts, but yeah, like you're saying, at the state level and for TA, it's a lot of rich, robust data. There's a lot, lot to come, I'm sure, with that, once that is available.

00:27:44.27 >> Sure. Yeah. And also reminding ourselves, too, you know, we want to accomplish everything, but taking it step by step, little bits, little bits at a time, and just you can't hit everything at once.

00:27:55.24 >> Yeah.

00:27:55.39 >> And reminding the districts of that, too, reminding them, hey, you know, we understand. Let's focus on one area and see what happens, and have that good conversation of what that could look like, moving forward. Because ultimately, we know their data is our data, and it's what we report, too. So we really need to know that they are supported when we have to do our own submission for the SPP-APR annually. Because without them, we can't meet requirements like we've done for the last six years.

00:28:22.74 >> Yeah.

00:28:23.45 >> So we know that importance, and we do share that with them, because we think that's a big deal. We know that's a big deal. So we have to do that due diligence to support them, or we're not going to be meeting requirements.  

00:28:36.73 >> Yeah.

00:28:37.02 >> Yes.

00:28:37.53 >> Yeah, it all works together, from the student all the way up to the state.

00:28:42.18 >> It does. Yeah.

00:28:44.20 >> Yeah. Well, Kristina, this was wonderful. You shared so many great tips and strategies, things I hadn't really heard about other states doing, so this will be great, I think, as folks out there are -- I know LEA determinations with DMS going on is definitely a really high priority in many states.  

00:29:05.91 >> Absolutely.

00:29:06.49 >> And I think they'll definitely pick up some ideas from what you all have shared, and all the great work you're doing in Illinois. So thank you so much.

00:29:14.49 >> Well, yeah, thank you so much for allowing me to talk about our LEA determination process -- 

00:29:18.82 >> Of course. 

00:29:19.09 >> -- and highlight the things that we are doing in Illinois.

00:29:21.70 >> Yes. Great work. Well, thank you again, and look forward to knowing and hearing about what comes next for you all.

00:29:28.63 >> Sounds great. Thanks.

00:29:32.89 >> A Date with Data is produced by the IDEA Data Center, which is funded by the U.S. Department of Education. Have a story about special education data that you'd like to share? We'd love to hear from you. Reach out to us at IDEA Data at Westat.com. To learn more about our Center and our work, visit us at IDEAData.org.