Artificial Intelligence, Genuine Results: West Virginia's Approach to AI

00:00:04.25  >> For the IDA Data Center, I'm Amy Bitterman, and this is "A Date with Data." Every month, I sit down with data quality influencers from around the country to share their stories about special education data and the work they do to improve outcomes for children with disabilities. Welcome to A Date with Data. In today's episode, we're diving into a topic that's rapidly changing how our work gets done: artificial intelligence or AI. We're joined by Jonathan Schenk, who is the Part B Data Manager from the West Virginia Department of Education. He's been exploring how AI tools can support a number of different data manager tasks and will share his experience. Let's get started. So, Jonathan, can you talk about why you initially started using AI in your work? Was there a specific challenge or task that prompted you to give it a shot?

00:00:56.75  >> Well, honestly, I started using it initially just because I was curious. The tool was available through our state's Microsoft Enterprise suite of office tools. So it was generally just out of curiosity, like, "What can I do with this? I have it." And it was sort of ... initially began with some low-risk experimentation because our office was fully staffed. There was a little bit of room to be able to do more exploratory-type tasks. Earlier on, I was using it to help me draft ancillary or supporting guidance documents that could be quickly reviewed and sent on for approval before publishing it to the state website or to share with LEA directors. And really at that stage, it was more of like a productivity enhancer rather than something to depend on, necessarily.

00:02:02.62  >> Sure.

00:02:02.87  >> And so that's how it initially started, drafting outlines of things or just maybe shaping, smoothing up, fine-tuning writing or something like that. So that's how it initially started. However, with staffing losses, like if we had some people in our office resign or retire, relocate, budget-driven risks, and if those positions weren't backfilled and then the workload remained the same. We still had to meet ... have the same responsibilities, same timelines. So the remaining staff, we had to absorb responsibilities that were previously taken care of by those who had left, retired or what have you. But again, deadlines, timelines, those all stayed the same. So as the human capacity kind of decreased, my use of this shifted from more of an optional tool to something that almost couldn't do without these days in a way. So that kind of led to more deeper experimentation, intentionally figuring out how to integrate its use into certain workflows and kind of stress testing the use of AI and this kind of new, less staffed environment that I found myself in. And really across both of those phases, the initial curiosity and experimentation all the way to incorporating it into my workflows, the core principle, all that, has generally remained consistent. So how I view it, AI is most effective as sort of a force multiplier for existing expertise and not really a replacement, because that is just as or more important than ever, that you continue to maintain the deep knowledge and expertise as a user of AI.

00:04:18.00  >> Yeah. That's a really good point that folks, yeah, shouldn't be thinking of AI as sort of a coworker that will take on these tasks that now are on your plate, but more of the support, that you use your expertise for developing the prompts, for carefully reviewing and revising and deciding what to use or not to use in terms of the results to really just help boost your productivity and efficiency. 

00:04:48.02  >> Yeah, that's exactly right. Because the AI can't be held accountable for things. Eventually, it comes down to: you're the one who produced the work, whether it was AI-assisted or not. So your own professional reputation and identity is essentially tied to that. So you have to be able to be confident in what product you're eventually releasing or publishing. If it was AI-assisted, it's going to come back down to that person who put that work out.

00:05:26.22  >> Yeah, and we know AI is not perfect. I'm sure you've ...

00:05:28.92  >> No.

00:05:29.22  >> ... had experiences where you've seen things that are not correct, missing from the results that you ...

00:05:34.60  >> Absolutely.

00:05:36.65  >> ... do end up with.

00:05:38.19  >> I do have a short little anecdote that speaks to that. So I was trying to quickly put together a CEIS, CCEIS, allowable cost resource. And I was using an AI tool that ... And it confidently cited an OSEP fiscal Q and A that I'd never heard of. So I asked it to produce the source, give me a link to it or something. And I spent time trying to find that resource on my own and ultimately confirmed that that fiscal Q and A that it cited in the output did not exist at all.

00:06:24.03  >> Mm-hmm. 00:06:24.62  >> So that little hallucination kind of raised my skepticism across the board and kind of reinforces the fact that the AI does not replace the subject-matter expertise. In fact, it depends on it because you, the user, are going to be the one who knows whether what the output is, whether it's correct, something is slightly wrong, very wrong. It comes back to the user.

00:06:53.72  >> Yeah. Can you talk about some of the ways, more specifically, that you've been using AI in your work?

00:07:02.33  >> Sure. So the way I use it personally follows sort of a consistent pattern. And that would be the sequence of uploading, prompting, reviewing, refining, and then using. So some concrete examples I guess I can put out there is, let's say, EDFacts data notes. So there's this document, "Writing Good EDFacts Data Notes," that I believe maybe NCES published that has examples of good, better and best data notes. So what I would do is upload that TA document as an exemplar or an example of what the AI should try to emulate, the writing, the quantity, the quality and so on. Then I would provide the rule-failure details, if I had something that needed to be addressed and the context surrounding that rule failure. And then I would ask the AI to draft a note matching that style based on the rule failure and based on any other context that I personally know that would be relevant to the state or the data. And then once it had that example to refer to, my input and the rule failure, it could generate something for me to review. I could verify it and make any changes, and then if it looks good and I'm willing to stamp my name on that data note as a submission, then I'll submit it. But it helps keep those consistent and clear whenever you provide that example for it to use whenever you're trying to constrain its output. Another example would be maybe like an LEA file review and corrective action plan analysis. So I could upload de-identified file review findings plus our state board special education policy. And then I could ask it to generate some cross-file findings and non-compliance tally which areas of which findings ... distribution essentially of the types of findings that were found, corrective action recommendations based on our state's policy in IDEA. And take what it gives me, review it for any regulatory accuracy before incorporating any of that into the monitoring report. Again, you as the expert in your own state's policy framework and IDEA itself, you have to know whether what the output is, is legitimate or acceptable or defensible or ... It's good for giving you a good starting point, and then you as the user have to use your expertise to review and refine before you actually do anything with it. Sure.

00:10:06.04  >> And something you mentioned, too, the de-identified piece, can you say a little bit more about that? And obviously, I know that's an important component is being really careful about what information you're putting into the AI.

00:10:22.48  >> Yeah, so I would never include any personally identifiable information in any prompt. So no student names, not even really IDs or any disability categories linked to identifiable individuals or any LEA-specific data without some kind of verified data protection agreement with the company that you are using the ... For example, we use Microsoft. So the AI that we use with the enterprise data protection is Copilot. So even then, I don't put in student names or IDs or anything like that. So whenever I'm including things in that, I would make sure that none of that is involved in the prompt. And a quick test I might say is, if this prompt were forwarded to somebody outside my agency, would it be a problem? And so if the answer to that is yes, then you need to revise it before you send that on through to the AI.

00:11:23.38  >> Okay. That's a great tip. Any other ways you've been using AI that you want to share?

00:11:30.62  >> Yeah, so I talked about that AI hallucination earlier with that fiscal Q and A when I was building that allowable cost resource. So in the process of doing that, the general workflow for that was our state ... I couldn't find any in the archives of files and documentations that we had at the state level, like a specific CEIS or CCEIS allowable cost document, so we needed one. So what I did was I looked myself at different state education agency websites to see what all was already out there. And so I collected some different examples of existing documents that states had already created. I uploaded all of those to an AI-assisted notebook tool. So an example of that might be Google's NotebookLM or Copilot Notebooks. And so what you can do with those is upload ... It just basically is a repository for a bunch of specific sources. And then the AI, it grounds all of its responses and outputs in only the sources that you put into it. So it doesn't take any liberties and pull stuff from out of the air or elsewhere on the internet. It's just based solely on what you put into it. So I used that to kind of synthesize a structured allowability table that was also grounded in 34 CFR 300.226 and 646, which are the CEIS and the SigDis regulations. So I made sure that those regulations specify what are allowable activities for those processes. So I used those sources to ground that AI as well. And so once it gave me a table of all the different allowable costs for CEIS and CCEIS based on that synthesis of all the other documents, I ran that table through different AI platforms, like Copod or Gemini or Perplexity. Since this isn't really super internal or private, personally identifiable information or anything like that, this stuff is already all publicly available in the first place. So in those kinds of instances, I might use other AI platforms to independently verify, just get different takes on it. And then once I had all that put together in a state that I thought was pretty decent, I sent that on to Cypher for expert review, and they actually said it was pretty good. So, it's, again, that last step. Before I put it out there and shared it with the world, I had it reviewed because I'm not a fiscal expert. I'm the ... If it's 618 data, 616 or what have you, that's different. But whenever it comes to fiscal compliance, excuse me, I felt like I needed to send that on to Cypher for a more expert-level review before I would publish that. And so, that's what I did with that one. 

00:14:54.63  >> Wow.

00:14:55.23  >> And I have one more example if you want to hear it.

00:14:57.30  >> Yes, please. Yeah.

00:14:59.61  >> So we in West Virginia had some new legislation enacted, West Virginia HB 2499. And part of that legislation was to ... It was a requirement for certain school personnel to be trained on certain aspects of our state board policy for special education and IDEA. And there were very specific discipline procedures, LRE and so on. So what I did was I copied that enacted bill text into AI and asked it to generate a training outline based on what that bill required. And I uploaded the relevant sections of our state code and our policy 2419, which is our state regs for special education. And it generated the training outline, and then I would have it ... Once it gave me the outline, I would have it fill out that outline section by section with the source-grounded content. So I wouldn't just say, "Create this whole training from scratch." Because if you give it such a large task right away, it may not do as thorough of a job. So if you do it by section of the outline, you tend to get better outputs. And then once I had all that stuff, I even went so far as to have it help me create a facilitator guide for that training, case studies because those can take time to really create on your own if you want to do a pretty good, thorough job with variety in the case studies. So it helped me make those decision-making flow charts. And, of course, all this stuff, I would review it, pass it on to other colleagues for review. And then that was that, and it ended up working pretty well.

00:16:55.74  >> Great, so it's been used now out in the field?

00:17:00.20  >> Oh, yeah. That stuff that was generated from that, specifically for that legislative training development, yeah, that was ultimately used. Of course, there were tweaks and some personalization I added. But, yeah, it definitely sped up the workflow of that quite a bit, as opposed to doing all of that manually and from scratch.

00:17:21.39  >> Right. Well, there's a couple tools I know you mentioned, like the Notebook. Are there other more specific tools or features of tools that you have found to be the most useful?

00:17:34.13  >> Well, I'm not sure that the ... So if you're using the Notebook tools, that's good for having it only respond or synthesize information based on exactly what you put in it.

00:17:47.97  >> Yep.

00:17:49.04  >> That's generally pretty good if you want to have just a repository of specific information. And that actually made me think of another use case, the DMS protocol completion for DMS 2.0. So in a Notebook tool, you could upload the DMS protocol template along with all of your own SEA's relevant documentation that pertain to that particular area of monitoring. And you could prompt the AI to answer each protocol question, citing those source documents and asking it to include citations of from which document those answers were based, review that output, refine it. And so if you're doing a task that requires integration of information from multiple sources and you need it to be completely grounded in only that information, Notebook tools might be the best way to go. But for general purpose tasks that are a little bit not quite that specialized, just any other type of just general AI tool, works pretty well. And I feel like the tools, they might matter a little bit less than the actual techniques that you use because a bad prompt in a top tier tool can still sometimes give you a bad output.

00:19:19.93  >> Yep.

00:19:20.63  >> And that's something else to consider.

00:19:22.98  >> Yeah. Garbage in, garbage out, like we say with data quality. Right?

00:19:26.57  >> Yeah, yeah. All the time. Yeah. It's universally applicable.

00:19:29.83  >> Yep. So what are some tips or tricks that you might suggest other states who are just starting to use AI might want to try to kind of dip their toe in this?

00:19:42.67  >> Well, I guess I would suggest that if you've never used it before and you want to experiment with it, try it out, I would probably start with something that you already know really well because that way you can be an accurate judge of whether the output is correct. I would not recommend starting out with your highest stakes work or something that you're not completely and entirely confident with because, as we were talking about before, your expertise matters just as much or more when using AI. So you don't want to give it the benefit of the doubt whenever it gives you an output and then you just say, "Looks good. We'll use that," because you have to know what ... You've got to know whether what the content is, is correct or not. I'd probably also say be very explicit when you're prompting. You got to tell the tool exactly what you want, what you don't want, what it shouldn't assume because oftentimes AI is pretty eager to fill those gaps unless you give it those boundaries to keep it in its lane. 

00:20:52.30  >> Right.

00:20:56.16  >> Probably also building reusable prompts. So what I've started doing is putting together a prompt repository. So it's just a simple spreadsheet with different columns, like the technique, what tasks it can apply to, how risky it is that the AI might hallucinate based on that type of content and different SEA examples of work, different types of tasks.

00:21:23.72  >> Right.

00:21:23.83  >> And so what that does, if you have a prompt repository with these reusable templates, it makes things more consistent. You can share that with colleagues. The AI use could be transferable to others in your office so that things are more consistent.

00:21:43.24  >> Yeah. So can you say a little bit about, for maybe for folks who aren't as familiar with AI, what a prompt is and just if you could mention a couple of the really critical, maybe, elements. You mentioned the boundaries and being explicit, if there's anything else you think is really critical.

00:22:01.02  >> So the prompt is basically what you're asking it to do or what you want. It's just your input to the AI. Your input is the prompt, and then what it gives you is the output. And so I guess what I could say ... I could give you a few examples of some techniques that I often employ whenever I'm using it. So one would be iterative refinement. So you start with a broad first draft of something. Then you can issue these targeted follow-up prompts or responses to adjust specific elements without having to start all the way over.

00:22:40.67  >> Mm-hmm.

00:22:41.24  >> You can ask it also to maybe proactively flag which sections it thinks I'll want to revise. So that's probably one of the ones that I use most often is just iterative refinement, getting some kind of output, reviewing it and then following up just like I would with a person who's doing a task for me in intervals. I'll ask them to do something. They bring it back to me. I'll review it, give them feedback and just kind of iterate on that and refine until it gets to a place where it looks good. You can use it as a thought partner or to spot blind spots. For example, rather than having it produce content, you can have it surface what you might have missed. If you have this plan or this idea, you could ask it to identify implementation barriers to that plan, things that you might not have thought of or alternative interpretations of that, what might someone else think of that or other perspectives you haven't considered. So you can kind of treat it as a friction-generating collaboration rather than just content generation. And AIs are definitely very eager to please, so sometimes it's good to ask it to tell me what's wrong with this or, "What am I missing?" because that can be useful as well.

00:24:14.80  >> Sure.

00:24:16.28  >> I think another thing might be is to ... One of the common techniques is to assign the AI some type of role or persona because when you do that, that can ground its response in the type of vocabulary, language and knowledge that that role might otherwise have if it were an actual person. So, for example, if you assign it a specific role, like a compliance officer or even an OSEP reviewer or something like that to anchor its expertise level and the tone that it uses in its response, or you can even do the inverse. So you could ask AI to critique your own draft as if it were an OSEP reviewer or something like that. You could say, "I'm submitting this as a slippage statement for the APR, for indicator, whatever. What might OSEP say in response to this? What am I missing? How could I improve this?" And again, your own expertise and experience, you know what OSEP might look for or what they require. If the AI ... Sometimes it'll tell you, "Hey. You may want to also tell you how or explain how you're going to do improvements within this statement." But you as a reviewer know that OSEP doesn't want further or future-oriented planning in the slippage statement. And oftentimes it's good to push back on the AI when it says something that you know is wrong, and you correct it. And it ...

00:25:55.36  >> Learns.

00:25:55.63  >> That works out pretty well as well. Yeah, absolutely.

00:26:01.02  >> Wow.

00:26:01.12  >> So yeah, the prompt is just your own input. The more specific, structured it is and the better output that you're more likely to get.

00:26:11.14  >> Right. And also, I like how you mentioned it's not really ... It shouldn't be generally kind of a one-and-done process. It's iterative. Don't expect upon your initial prompt, you're going to get exactly what you want and that it should be sort of a back-and-forth, a cycle over repeated questions and reviewing of responses before you kind of end up with what you are happy with.

00:26:38.12  >> Yeah. Yeah. It's almost never a one-and-done. And especially for work that is going to be either published or released some other way. If you're just using it as a thought partner, you know, that's a little bit more informal and just you can basically converse with it as if it's a person to help you spot issues or something like that. But yeah, if you are trying to create an actual product, definitely it would never be a one-and-done. I would say no.

00:27:08.16  >> Yeah. Okay. Anything else that you want to add?

00:27:13.68  >> I would say that if you're not careful, the iterating and the refining can ... You have to know when to stop because you can refine and refine and refine over and over and over. But eventually, you're going to want to kind of set a limit for yourself or just ultimately understand when it is good enough.

00:27:38.44  >> Sure.

00:27:40.75  >> Good enough not as in just fine, but it's never going to be the absolute most perfect thing that could possibly ever be created. So you just have to be realistic about when enough is enough for refinement because you might find yourself doing that for half a day if you're not careful. So just go in with some expectations and a general limit for refinement because it can snowball pretty quickly. 00:28:12.73  >> Yeah. It can be a rabbit hole, and the AI will keep going as long as you keep going. It's never going to ...

00:28:17.45  >> Oh, yeah. 00:28:18.06  >> ... tap out and say, "I'm done. I've given you all that I can give you."

00:28:21.33  >> Right. It almost always ends with, "If you want, I can help you do this, this, this, or this."

00:28:24.59  >> Gives you all the suggestions.

00:28:25.27  >> "Just say the word."

00:28:26.10  >> Sure. 00:28:26.29  >> Yeah.

00:28:26.67  >> Yeah. All right. Well, I think that wraps up today's episode. Thank you so much, Jonathan, for sharing how AI is being used in your state and offering some of those really great practical examples and benefits. Really appreciate you being on, and we will meet again next time for another Date with Data.

00:28:47.18  >> Yeah, not a problem. And one last thing I might leave you with is you can actually use AI to learn AI. So if you have a prompt or you're not quite sure how to word what you're wanting to do, if you tell the AI, say, "I'm wanting to be able to do this, this, and this. What would be a good prompt to get the kind of output I'm looking for?" And it can help you apply those constraints and roles and that kind of stuff. So use AI to help you get better at AI.

00:29:21.08  >> All right. Great. Well, thank you again, Jonathan, really appreciate it.

00:29:25.65  >> Not a problem.

00:29:29.60  >> A Date with Data is produced by the IDEA Data Center, which is funded by the U.S. Department of Education. Have a story about special education data that you'd like to share? We'd love to hear from you. Reach out to us at ideadata@westat.com. To learn more about our center and our work, visit us at ideadata.org