Plan Ahead, Perform Powerfully: Developing and Submitting a Successful SPP/APR

00:00:04.30 >> For the IDEA Data Center, I'm Amy Bitterman, and this is A Date With Data. Every month, I sit down with data-quality influencers from around the country to share their stories about special education data and the work they do to improve outcomes for children with disabilities.

00:00:21.42 >> Welcome to A Date With Data. On this episode, we are going to be talking about writing a high-quality SPP/APR. I am joined by Jeff Phenicie, who is the Executive Director of Data, Monitoring and General Supervision with the Virginia Department of Education. I'm also very excited to introduce Rachel Wilkinson, who is an IDC TA provider who's been very involved in all of the support that IDC provides states regarding their SPP/APRs. Jeff and Rachel are going to be sharing tips and tricks for developing and submitting a high-quality SPP/APR. Thank you both so much for being on. Jeff, can you start out by telling us a little bit about your role at the department?

00:01:04.04 >> Sure thing. As Amy mentioned, my name is Jeff Phenicie with the Virginia Department of Education. I'm currently the Executive Director of Data, Monitoring and General Supervision. I oversee the offices that has all of the state monitoring in it. I oversee the office that has all of the data collections, December 1 child count, APR and such. Then I have a third office that I oversee that kind of the catchall office, so for the special projects. I've been with the Virginia Department of Education for 21 years now in various roles, and I have been working with the SPP/APR since the 2004-2005 submission. I've served as the role of the APR coordinator until this year's submission where I'm transitioning those responsibilities over to the next generation and build some capacity.

00:01:58.75 >> Great, so Jeff has been around quite a while and been working on the SPP/APR for 20 years, part of the reason we brought him on to talk about his experience and share some of his strategies. So thank you. Can we start just talking about, what has your process been for putting together your SPP/APR every year?

00:02:20.98 >> For me, it's always been about the preparation.

00:02:23.27 >> Mm-hmm.

00:02:23.82 >> If we've done the prep work throughout the year, the SPP/APR really writes itself. It truly is the easiest part of the process. There are just lots of moving parts. If at any one point, there really are three different APR submissions that overlap each other. We're closing out the noncompliance findings from the last APR, ensuring that all the findings of noncompliance have been addressed, properly documented in accordance with OSEP QA 23-01 and such. We're gathering everything we need for the submission of the current APR, items such as opening and closing data-submission windows, requesting data from other units across the Department of Education. We have staff validating self-reported indicator data and so forth, and we're already prepping for the next APR. We're in the process of drafting the notification letters to inform superintendents of noncompliance identified in the current APR that will be reported out in the next APR. At this point in my career, I really envision myself as the head coach. I'm not only ... Not only does the head coach develop the game plan, but coaching staff must ensure everyone has a clear understanding of their roles in the big pictures. What appears to be inconsequential to one can trigger the domino effect and have lasting consequences that may take multiple APRs to rectify. With the introduction of the Indicator 18, the team got bigger this year. Now, I need to ensure not only staff in my offices know their roles and responsibilities but staff in the Office of Dispute Resolution know their roles and responsibilities. Specifically, are they properly documenting noncompliance findings with specific regulatory citations on the front end, and are they following up with timely closure? To stay with the coaching theme, if one teammate misses their assignment and it goes unnoticed, you may not have what you need when it's time to write the current APR, adding an additional level of stress that can easily be prevented with proper planning. You also need to have a game plan for those unexpected items. Have redundancy built in. If a key staff member unexpectedly goes on medical leave, what is your plan? I've learned over the years to not work in a silo but build a team. If you don't have a team, lobby your senior leadership for one.

00:05:00.38 >> Great. Yeah, those are some great, great thoughts. I know there are some states where it is very siloed, and maybe one or two people focus on the SPP/APR, so definitely creating that team approach seems like a great strategy and has been working well for you, so thanks. Indicator 18, and that's new for this year, and there's also some new questions that are in the introduction. Can you talk a little bit about how you have prepared to respond to those new questions in the introduction as well as Indicator 18?

00:05:34.84 >> Well, I believe the new questions in the introduction are really OSEP's attempt to ensure full implementation of Q and A 23-01 sooner as opposed to later. Rather than waiting for a full round of DMS 2.0 to cycle through, they're just gathering all that information up front. Much of the same information required in 23-01 is outline in the protocol, the general supervision protocol, are now gathered in the introduction. So if you've already gone through DMS or if you have revised your policies to be in alignment with 23-01, the new eight questions really can just be lifted from your general supervision policy manual. That's the direction we're going at this point in time.

00:06:19.76 >> Yeah. That's a great tip for those especially who have been through DMS already and have that information prepared, and those who are in the process of preparing or haven't gone through it, you're going to need to do it anyway, and for the SPP/APR and for DMS, so kind of multi-purposing all of that information is a way to be more efficient.  

00:06:39.68 >> Okay, and I guess let's talk a little bit about Indicator 18. So really, just what goes around comes around. Indicator 18 is new in this current format, but it feels very familiar. Up until around, I think it was FFY13, 2013, we had a very similar indicator known as Old Indicator 15. The process was the same. Just like with the other indicators, preparation is key. As I previously mentioned, cross-office communication is more essential with this indicator than any other. It comes down to documentation, whether the noncompliance is found through your cyclical monitoring, your targeted reviews, the APR compliance indicators, fiscal monitoring, complaints, due process. It all just comes down to documentation. Are we clearly identifying the findings of noncompliance by specific regulatory citation? Are we properly documenting notification of noncompliance and keeping accurate records of individual and systemic correction in a timely manner? It all takes planning across the units. You are going to have noncompliance, whether it's just five related items or 5,000 related items. Proper documentation for notification to timely closure and everything in between is your focus. In most instances, you and your staff control your own destiny. If you're aware of looming deadlines, you will get what you need, but if one of those things fall off the radar, that's what's going to come back to burn you.  

00:08:18.66 >> So can you share any tips of strategies that you have gathered over the years of working on this that might be helpful to share with newer staff? Maybe it's their first time writing in or pulling together the SPP/APR. What are some ways you can sort of guide them to help think about making sure their SPP/APR is of high quality? 

00:08:41.76 >> First and foremost, it always goes back to the measurement table, for me. I'm in my 20th year of doing this. That is the document that's your grounding source for information. If you're unsure of what's asked in the indicator, just go back to the measurement table. Answer what is asked exactly, no more, no less. And use the resources provided to you. The National TA Centers are there to assist. That's definitely, as you know, Amy, one of my go-tos. They provide you with physical resources, technical assistance from extremely knowledgeable staff that have been in your shoes. Whether it's a monthly TA call, an office hours, an in-person conference or one of the countless physical resources that you guys provide, the TA Centers are there in this unique situation of, first, hearing firsthand from OSEP and then seeing how multiple states address those requirements. I just can't express enough how beneficial the TA Centers' staff are and able to assist you. I found one of the most beneficial resources that you guys provide is that brokering connection with like staff in other states. Connecting me with my colleagues from across the nation can be so beneficial. Sometimes you feel like you're on an island within your state, but you're really not alone. Other states are wrestling with the same exact issues. The key is finding out who, and your IDC contact or other National Center leads are the keys for finding those staff. Utilize them. You guys are in the TA business, so just utilize their expertise.

00:10:32.73 >> Great. Thank you, and that's such a great segue into Rachel Wilkinson, who is an IDC TA provider. So wondering, Rachel, if you can talk about, from your perspective as a TA provider who has worked with many states, reviewed a lot of states' SPP/APRs, what are some of the things you might recommend for writing a high-quality SPP/APR?  

00:10:57.80 >> Sure. Well, thank you, again, for having me, and again, I'm Rachel Wilkinson with IDC, and I have the pleasure of reviewing a lot of SPP/APRs every year, and there are lots of things we see that states do extremely well, and there are areas of challenge that I think apply to everyone. So some of the things that stand out to me, thinking of looking at this information from a reviewer perspective, stakeholder engagement is always an area of challenge that I think states have because it is such a large section that's really getting at more than just target-setting, which is mostly what the focus was a few packages ago with the SPP/APR and now has branched into how to get diverse audiences engaged with information that goes beyond target setting but also includes looking at improvement strategies, evaluating progress and analyzing data. So one of the things we always highlight is showcasing the ways in which you engage with stakeholders as state staff or community partners, and that might mean focusing on the activities you do to build diverse stakeholders' capacities to learn. So maybe it's putting together resources or PowerPoints or trainings at your state advisory panels or if you travel regionally to different school districts and work with different staff that way. That can be extremely powerful to showcase. And to also focus on what you do in the reporting period, SPP/APR is unique in that it kind of straddles multiple reporting periods when you're writing it versus when it took place versus when some of the data is pulled in. And so it's important to just focus on the activities that were in the reporting period, so the FFY2023 SPP/APR reporting period will be the 23-24 school year, so focusing on the activities in that period and then how, again, you increase capacity of diverse stakeholders or community partners. And then another thing that we see is baselines. Sometimes it's hard to know when to reset a baseline, so some states might look at their baselines, and think, "Gee. We haven't updated this in 15 years." And that's fine because baseline should really be updated when there's a change to either the data source or the methodology that makes the data for this reporting cycle no longer comparable to any of the prior years. So it's always important to think about, has my data process changed, maybe for Indicator 1, for graduation rate? There's new graduation rate requirements. Will that have a meaningful impact on the data source or methodology of my indicator, which would require me to reset a baseline? Or if I switched in the case of, say, Indicator 8, from using a census survey to sampling or vice-versa, in which case that could also have an impact on your baseline data and its comparability. Representativeness and non-response bias are two of the most popular items that we have come up in terms of questions during the review process, and we have some great staff who really support that work. So I always would just remind people, at the core, there are both two different things, measuring similar but different things, so they should be treated separately. Sometimes we see the same language repeated for representativeness in the non-response bias field, but really that's not quite accurate, and so you'll be missing some key components if you do that route. So you'll just want to make sure that you're really differentiating the two, appreciating the connectedness but not that they're identical. Then for Indicator 18, it's really calling this into account a little bit more than maybe in prior years, but correction of noncompliance has always been the cornerstone of the compliance indicators and 4A, which is technically results indicator, but it's kind of always placed in a strange gap between the results and compliance indicators because it's similar to both. It has components of both, rather.

00:15:19.55 >> Mm-hmm.

00:15:20.10 >> So the correction of noncompliance piece, you really want to articulate what you did as a state to verify and address the findings so that you know they're corrected, and you don't want to use maybe more boilerplate language, but your general correction process. Instead you want to be focused on those specific findings for the specific reporting period and how you exercise your general supervision responsibilities to verify those findings were corrected. And then the last piece I'll mention, having now looked at some SPP/APRs that have been coming in for this review cycle with new introductions, people are doing a great job sort of amassing all the information that is asked. A lot of it was technically in the SPP/APR before, but it might be as simple as moving some information around to different check boxes or different text fields. But there is some new components that do need to capture information, so thinking about what should be in the fields in the introduction and aligning it to the compliance indicators as well as to Indicator 18, so how findings are reported, what your monitoring processes are in your data sources, how records are selected for review. If you're going to mention that in the introduction, in those required fields, as well as in your indicators and Indicator 18, really make sure there's consistency across all of those. That's really what OSEP has been pushing for, is internal consistency that's reflected in your process.

00:17:00.31 >> Great. Well, thank you so much, Rachel. And one of the things you mentioned, I think one of the first things, a challenge that state sometimes encounter, is that engagement of stakeholders and going beyond just including them as part of the target-setting process but also analyzing data, improvement strategies and evaluating progress, and I know that's something that, Jeff, you and Virginia have been really trying to make sure happens every year meaningfully and authentically. So can you talk a little bit about ways that you have been able to kind of continuously engage stakeholders really beyond the target-setting aspect?

00:17:38.46 >> I think the key word in that is "continuously" engage stakeholders. 

00:17:42.58 >> Mm-hmm.

00:17:43.26 >> Effective stakeholder input is not a one-and-done, and that was a lesson that took several years for us to figure out. We would have those culminating meetings in December every year and say, "Here is all the" ... And it would be like an 8-hour meeting with just a roomful of people, and it just ... It was really not effective. What we have found out to be the most effective is that continuous stakeholder engagement throughout the year. So in Virginia, we have a structure in place that I'm sure isn't unique. We have an APR coordinator that kind of oversees the entire process, but we have indicator chairs organize topically such as the secondary indicators, so 1, 2, 13 and 14 are grouped together.  

00:18:30.53 >> Mm-hmm.

00:18:30.80 >> The early childhood indicators like 6, 7 and 12 are grouped together and so forth. And those chairs meet throughout the year with their specific stakeholders that they engage month in and month out over the course of the year.

00:18:51.28 >> Mm-hmm.

00:18:53.46 >> This could be done at specific meetings, conferences and so forth. So last week, I was up at one of our state colleges. We were meeting with 200 stakeholders across the various areas, so we were able to gather some stakeholder input at that event. They're analyzing data. They're evaluating process. They're making ... It may result in revisions to targets, improvement strategies and so forth.

00:19:21.44 >> Mm-hmm.

00:19:22.39 >> So what we've learned to do is to gather that in a questionnaire format, for lack of a better word. It could be a static document that gets printed out. It could be a fillable PDF that gets typed on and then e-mailed back to the indicator chair. Or some of the indicator chairs have converted it to a QR code so they can access it through their PowerPoint through a flier or so forth and then provide input that way. One thing that we found in this questionnaire to be most beneficial is it provides that equal voice for all stakeholders. Often when you put a group of stakeholders together, you have one or two big personalities, and the kind of dominate the room. And they think they're speaking for the entire group. So this questionnaire, that gets collected at a later point in time, whether it's at the conclusion of the meeting or a week or two later. Some of the best suggestions that we get come from those that are quietly sitting in the room, listening, and then they provide their input at a later time after they can post their thoughts. It just seems to be a very beneficial tool for us, and as a side note, it also just provides solid documentation that we do have and track meaningful stakeholder input, so it's a win-win for us.

00:20:49.56 >> Yeah, and just offering those different mechanisms to gather that feedback, like you said, because some personal might be more willing and more able through writing more anonymously versus speaking up at meetings, so just giving all those different opportunities, I'm sure, allows you to, like you said, just gather a lot of rich input that you've been implementing. So great strategy there.

00:21:14.89 >> Yeah. Another way that we've been dealing with that, same type of ... You can use, actually, the same tool. 

00:21:21.07 >> Mm-hmm.

00:21:21.39 >> But the indicator chair can kind of summarize the meeting also.  

00:21:25.05 >> Mm.

00:21:26.17 >> And just use the form as a summary document in addition to those individual ones that come in but things like they can, on the document, tell us who is in the room, the roles that they played while they were in the room, and really also gather the number of parents because that was one of the new things that came out of the last [Indistinct] was the actual physical number of parents that were engaged throughout the entire process. So that was another way to gather some of that information also and then be able to document that X number of parents were involved in this process at this particular event.

00:22:06.37 >> Yeah, keeping track, like you said, along the way, not waiting until the end and being like, "Oh, how ... Let's go back 6 months. How many people were there, and how many parents?" and figuring it out from the past, really very explicitly making it part of your process along the way for everyone. Jeff, any other suggestions you have you want to mention that you haven't already talked about? 

00:22:30.44 >> Don't stress. That's the biggest thing I can tell you, is just don't stress over the process, and utilize OSEP's clarification period for what it was intended for. Make sure you do ... you schedule that meeting with your OSEP state lead, and if you don't understand their written comments, ask for clarification at that time. The state lead is not going to tell you what to write, but they definitely will provide you with what they believe is missing. It's basically a penalty-free redo, so don't stress over something that may have met the standard last year. I'm sure it's going to be frustrating because you may have written the exact same thing the previous year, and it wasn't identified as a shortcoming or so forth.  

00:23:15.76 >> Mm-hmm.

00:23:17.63 >> But although it can be frustrating, it could be different readers providing different interpretations of what they're looking for. It could be a new focus for OSEP that year. Who knows why they're asking for it? But just don't take it personal. Don't stress. It's not your APR or any reflection of your abilities. It's the state's APR period, and just go with that mindset, and you'll be fine. 

00:23:44.79 >> Great. Well, thank you so much, Jeff, for all of your wisdom and just having such experience in this area, hearing your suggestions that you've been using for many years. I think it's so helpful probably for other states out there. And before we wrap up, I'm wondering if, Rachel, can you just talk a little bit about the support that IDC provides states? Jeff mentioned a few of the things already, but other areas where states can reach out to their IDC state liaison, and we can help you with your SPP/APR.

00:24:18.70 >> Absolutely. Well, first, I hear Jeff say that he liked connecting with other states, so bothering Jeff and getting support from him to help you not worry would be an excellent resource. I say that jokingly but also semiseriously because Jeff is tremendous, has a lot of expertise and is always available to provide support in such helpful ways. But IDC is also here to do that and limit that responsibility on Jeff, of course. So one of the things that we do that's probably the most popular IDC service is the SPP/APR reviews. So we do offer that as a service that begins on December 1st and continues until the week before the February due date for the SPP/APR. And we conduct a pretty robust process where there's a first reviewer, a second reviewer, and then we have a core reviewer who kind of consolidates everything to make sure it all looks good and that you're getting the most thorough and accurate, clean response that you can. We do take great pride in this work, so we definitely try in earnest to turn around our reviews within 10 business days and to support states with information, insights on what they need to consider in their SPP/APR, what's working really well, where there could be opportunities to improve and any concerns we might have just based on information that maybe is there but doesn't seem quite right or isn't there and should be. So that is a great way to have an extra set ... well, three extra sets of eyes on your reviews before you finally submit them. So I always encourage people to use that resource and service. We also have an SPP/APR data quality peer group as well as an indicators 4, 9 and 10 data-quality peer group. So obviously both are heavily focused on the SPP/APR as a whole, and then the latter is far more focused on those three very unique indicators. I plug that because I am lucky enough to be one of the co-leads for that group, but it is also an excellent resource to learn from your peers because I do think, to Jeff's point, that's one of the most valuable ways to learn, is from others who may be in similar states, maybe experiencing similar challenges. And so both groups are a chance to debrief with your peers and talk through some common trends, challenges, experiences and support each other. Of course, you can always submit questions, concerns to your state liaison. Our state liaisons are great at working internally to find the most experienced staff who are internal within our organization to support you with a response. So if they need more support, there's always more support waiting in the wings so that we get you the most thoughtful and accurate response possible. And then annually, we have our SPP/APR summit that's also a great resource to use and attend. We usually allow around two state staff members from each state to attend and get information about upcoming submissions, things to do to prepare more broadly, hot topics, so of course, Indicator 18 and the introduction. Some of the changes to Indicator 4 were a big focus of this last summit. But we do try to tailor it to what we hear from the field, what is needed most. So that's a great resource to also use and to attend if you're able to because it will give you sort of the overall view of the SPP/APR as well as drill down into more specific areas. So those are just some of the resources and services that come to mind.

00:28:09.85 >> Great. Thank you so much, Jeff and Rachel, for being on. Hopefully states out there listening picked up a tip or a trick that they can apply as they're working on their SPP/APR, which, as we discussed, is really a yearlong process. It is crunch time in the holiday, December, January time frame, as they're due early February. But just according to one of Jeff's tips is, be thinking about it all year long, planning for it, gathering that information, and don't wait until the end of the year. So thank you both for being on.

00:28:44.17 >> Thank you.

00:28:45.40 >> Thank you much.

00:28:48.46 >> A Date with Data is produced by the IDEA Data Center, which is funded by the U.S Department of Education, have a story about special education data that you'd like to share. We'd love to hear from you. Reach out to us at ideadata@westat.com to learn more about our center and our work. Visit us at ideadata.org