A Disciplined Approach: Strategies for Developing More Meaningful Discipline Data

00:00:04.25  For the IDEA Data Center, I'm Amy Bitterman, and this is A Date with Data. Every month, I sit down with data quality influencers from around the country to share their stories about special education data and the work they do to improve outcomes for children with disabilities.

00:00:21.86  Welcome to A Date with Data. I'm your host, Amy Bitterman, and today we have a very special episode where we're going to be diving into discipline data quality. We're going to talk about questions like why is high quality discipline data important? How can better data lead to better outcomes?

00:00:38.06  And what are some strategies that states and districts might want to consider to help improve their discipline data quality? We're unpacking all of that and more with our two guests. We are joined by Heather Reynolds, who is a technical assistance provider with the IDEA Data Center, and Dr. Kent McIntosh, who is co-director of the Center onPositive Behavioral Interventions and Supports. Heather and Kent, thank you both so much for being here and sharing your expertise with us.

00:01:04.70  Thanks, Amy. Glad to be here.

00:01:06.74  Likewise.

00:01:08.26  All right, so Heather, can you kick things off by just describing some of the common challenges that you've seen states and districts experience when it comes to collecting, analyzing, reporting, and using high-quality discipline data?

00:01:23.56  Sure, Amy. I think some of the challenges are just like challenges around any sort of, you know, data collection and analysis and reporting, you know, things like, is the data entry reliable? But some things are also a little bit different for discipline data. I think one of the things we've had states talk about a lot is, you know, maybe folks are not defining or categorizing data in the same ways. And so that can lead to some inaccuracy. Mostly, I think the question that we've asked a lot when we’ve talked to states about this is, you know, do your data really reflect experiences that students are having in your buildings day to day? Are your data really reflecting or representing the kinds of experiences that students have around discipline?

00:02:19.87  Yeah, Heather, you make a really good point about that. You know, behavior can be so subjective and those categorizations are there. And so it's really important that people have this understanding that, you know, behavior to one person is different to somebody else. And, you know, when we look at discipline data, when we look at referral or suspension data, it's really important for us to remember those are actually counts of adult behaviors, not student behaviors. So when there is a disruptive behavior that happens, there needs to be an adult to see it.

00:02:58.67  They need to categorize it as, is this referable or not? And sometimes they might even say, is it worth me writing this up? People might use these data for different purposes. Somebody may want to just start writing students up over and over, to give them, build up a little bit of a case for different kinds of services. And some just think, you know what?I can handle that behavior in my classroom. I can deal with it. I don't need to exclude students. So we just have to recognize, to look at it with a little bit of recognition that those data are squishier than some of the other ones out there.

00:03:39.88  Yeah, that's a great point. And that subjectiveness piece, which we similarly, don't see necessarily in other data, like if you're talking about exiting data whether someone either graduated or didn't graduate, there isn't sort of that room for interpretation. So Kent, given these challenges that we know states and districts face, what are some strategies that they can use to help improve the quality of their discipline data? Do you have any examples that you've seen with states or districts?

00:04:09.26  Really, really good question. I think of two big recommendations that I've got. So we at the University of Oregon house a discipline data web application called SWIS, the School Wide Information System.We operate that at cost, and I don't make a dollar from it. But one of the things that's really useful is when we have research findings, we are able to turn that right into the application because we have our programmers, developers are right in the building. One of the things that we have in SWIS that we highly recommend for anybody using it or not, is having really clear operational definitions of each of the behavior categories.

00:04:51.72  What is worth writing up? What is a major? What is a suspendable category? Because, you know, so many of these, as Heather and I talked about before, are subjectively defined. We want to be really, really clear when somebody has an understanding of it. The data are better because of that.

00:05:10.86  You can go on. To the SWIS website and download completely free whether or not you're using it, the operational definitions of behavior categories that we recommend and that we use, like I said, regardless of which ever application you use. So that's number one.

00:05:30.67  And then number two is trying to make the data entry as efficient as possible. So sometimes discipline data are these just sort of blank pages of, you know, comment fields or open text fields are like describe happened in your own words. And when it comes down to it, that is longer to write, it's longer to enter into the system, and it makes it far harder to actually make good decisions from it. So I highly recommend using dropdown menus, making sure that each one of those fields is actually useful and efficient. Using some kind of system that's designed from the standpoint of somebody actually entering that information. And so sometimes we'll have people who use, you know, it's nice to have an application where you can either write it up on paper or you can enter it right into the system, whether that be on a computer or on your phone. Any way where data entry is efficient is going to make it more likely that that information is actually recorded.

00:06:44.42  I think, Kent, that's a great point around having really efficient processes. I think one of the things we've also heard states talk about is once you've developed those efficient processes and procedures, being able to make sure that training's available to everybody that's involved in the collection and reporting of the discipline data. You know, I've heard states talk about maybe on-demand training that they've posted somewhere so that folks from, you know, the person at the school level who's entering that discipline data all the way to, you know, maybe somebody at their central office or their state level who's doing some validation of that data, can see what that process is meant to look like. And Amy, as you know well, we talk a lot about clear documentation at IDC. And so some of those tools like data dictionaries, which is really kind of what Kent was talking about, right, around the SWIS documentation, can ensure that everybody's talking and thinking about things in the same kind of way.

00:07:51.82  Another strategy that I've heard states talk about a lot is making sure that they're working across departments. So a lot of the folks that we work with at IDC, of course, are in that special education department or world at their state ed agency. And we know that discipline data, while there are requirements about, you know, collecting and reporting it for students with disabilities, certainly is a thing that happens outside of special education. And so having conversations with folks across your agency to ensure that everybody across your whole agency is really talking and thinking about and using those same definitions for discipline data has been really important in a lot of states.

00:08:38.74  Great points.

00:08:40.53  Yeah, those are some great tips. And I'm hearing things like consistency seems to be a common theme, making sure that there are these shared definitions that are being used across, like Kent's talking about the drop down fields where everyone is kind of forced to select something from the same list rather than these more qualitative description type responses that someone has to kind of go back in and potentially code. And also just the training piece is so important too. So thank you for sharing those. Heather, can you talk about really why is it important for states and districts to have high quality discipline data? We'll talk about, you know, the need for it and what are some ways to achieve it, but why does it really matter if your discipline data are high quality?

00:09:33.51  Well, we know that data use can support good decision making. It can help point out differences and start conversations about what's happening and how it can be changed or improved. But just like with any other type of data, you can only do that with discipline data if it's of sufficient quality that you really have confidence in it. To me, that's one of the main reasons to do some of that other work we talked about, right? The clear documentation, the on-demand training, you know, having those conversations across departments really help you ensure that you're getting data that are of high enough quality that they'll be useful for decision making, because ultimately that's what helps you create change and improve outcomes for students.

00:10:21.60  Yeah, that's a huge point. I think that, you know, so often if we're just sort of thinking about what's the task in front of us and we're so busy doing it, a lot of times we might just think, you know, what are the data that we need to submit to the state? What are the data that we need to submit to the feds? Instead of saying, like, how are we actually going to use this? And a lot of the work that we've been doing from the center and associated research projects over the last 15 years or so is helping teams look at their discipline data.

00:10:51.50  And instead of just saying, is it going up? Is it going down? Is it disproportionate? Is it not? Using these things called precise problem statements. And so a basic problem statement is something like, Oh, my gosh, referrals have gone up like crazy in our middle schools in the last month. A more precise one might say how much of an increase, a double in the amount of referrals.

00:11:18.19  And it looks like the focus, the big change has been in hallways or common areas. And it looks like it's a bigger deal on Wednesdays when there are early release days or something like that. Then all of a sudden when we've got those precise problem statements, it's really, really helpful because we can just put on our teacher problem-solving hats and say, okay, here's the situation. Here's what we can do about it. Instead of just admiring the problem or instead of just logging the information and going on gut feelings.

00:11:53.64  So there's this whole approach called team-initiated problem solving or TIPS and you can find it on PBIS.org website that guides teams through this process of taking this almost hunch-based information and turn them into statements that can be really, really helpful in drilling right down into the problem and then being able to monitor whether the things that we're doing are making things better, making things worse, or not making a difference.

00:12:23.56  So really helping taking that information and if you have that high quality data, hopefully being able to really use it to do some root cause analysis, to drill down, make those types of improvements that we all are hoping for by having that good quality data at our fingertips.

00:12:43.11  Absolutely.

00:12:44.60  So Kent, what are other data that you recommend or have seen states and districts look at kind of alongside their discipline data and why? How does that really help them better understand some of those root causes and ways to make improvements?

00:13:01.78  Discipline data give us one view of what's going on in terms of student behavior, in terms of student outcomes. So there are so many other areas, especially that states and districts have invested in that are incredibly helpful. So obviously, looking at something like achievement data can be helpful, too, because of the links between academic and behavior performance. But also looking at attendance, looking at chronic absenteeism, and also looking at measures of school climate and school safety.

00:13:37.27  So being able to ask students and staff in buildings and family members, what are their experiences with the schools and what are their experiences with their schools? Behavior support systems is incredibly useful. You know, the extent to which students feel safe, the extent to which students feel like there is an adult that they can trust with a problem if they need an adult to help solve a problem. All of those things, you know, the extent to which they feel like they belong, extent to which they can feel like they can show up and be themselves in school. You can't really get that information just from discipline data. Really asking students themselves is a critical component.

00:14:26.90  And also just looking at other information. Some of our schools do great work looking at counselor and nurse visits, trying to get a different sense of what information is already out there that can help with that, as you were saying, Amy, that root cause analysis.

00:14:44.25  And then the last thing that I would recommend is looking at fidelity of implementation of some of their initiatives. So that's basically the extent to which the adults are putting into place the practices that the state or district or school has agreed on doing.

00:15:04.70  And so if we aren't measuring how well we're implementing implementation of an SEL program, implementation of PBIS, for example, then we're not going to have a good sense of whether we're doing what we said we would do or whether it's actually having an impact.

00:15:27.27  Yeah, thank you. I think that is a piece that sometimes gets forgotten that implementation fidelity component. And also, you know what you were saying that discipline data is just, you know, one important piece of the puzzle but shouldn't always, you know, be looked at in isolation. You need to look at a lot of other data and how it all fits together and what might be related to really try to get some change happening. And Heather, any other data you might recommend states, districts look at alongside their discipline data?

00:15:59.13  I think it depends on what you're seeing sometimes in your data. I feel like sometimes folks are not quite sure where to start in determining like, oh, we need to also look at attendance or outcomes those things, you know, so sometimes even starting with things like. Do we see any differences in the data that we typically look at? Achievement, graduation, attendance between students who've been suspended and those who haven't.Between students with disabilities who've been suspended and their peers maybe who have also been suspended or not.You know, even starting to disaggregate some of the data can help us think about things that we might want to investigate further. And as you mentioned before, Amy, that root cause analysis process or that process of really thinking about what’s underneath the data that we see here can be really helpful to schools districts and states and thinking about, you know, what are some things that we might try? What are some things that we might do a little differently to improve outcomes?

00:17:13.51  Thank you. Well, I think that's a wrap on today's episode. Thank you so much,Heather and Kent for sharing your valuable perspectives and really hope that folks listening, states, districts, that some of this information has sparked ideas as you continue to work on improving your discipline data and improving outcomes for your students. Thank you both so much.

00:17:37.55  Thanks for inviting us.

00:17:38.90  Yeah, thanks, Amy.

00:17:42.43  A Date with Data is produced by the IDEA Data Center, which is funded by the U.S.Department of Education. Have a story about special education data that you'd like to share? We'd love to hear from you.Reach out to us at IDEA Data at westat.com. To learn more about our center and our work, visit us at ideadata.org