Ep. 351 Why Federal AI Needs Trust, Governance, and Human Oversight!

Speaker 1  0:04  

Hey, John Gilroy here. Today we’re going to talk about artificial intelligence, the federal government, and the human in the loop. Hit the music manny. Welcome to the Federal Tech Podcast. My name is John Gilroy, and I’ll be your moderator. Our guest in the virtual studio today is Reggie Towson, Vice President AI Ethics, Governance, and Social Impact at SAS. Reggie, how are you?

Speaker 2  0:28  

I’m doing well. I’m happy to be back, John.

Speaker 1  0:31  

Want to talk about these peculiar individuals, these humans? I mean, we got to talk about the humans and AI, and let’s not dazzle people with Chat GPT 6o and and all kinds of of different terms that is floating into the world here, but I want to bring up the human element here and and talk about a report that has just come out, and I will put this report in the show notes for our listeners, and the report is SAS came up with the report and they have some insights supported by IDC, and the title is Data and AI Impact Report: The New Economics of Trust. So, so here’s the start: the economics of trust. Tell us about this report.

Speaker 2  1:10  

Well, it’s really pretty straightforward, and I’ll break it down like this: Trust matters, right? And it’s it’s it’s in a duh moment, but you know, trust drives business outcomes, and what we do in the report is we start to suss out what happens organizationally as it relates to trust and trustworthiness when AI shows up. And one of the things that the report reveals is AI as a research tool, basically the way we’ve been using it in the generative AI era has a certain level of trust associated with it, but what we’re seeing is that once when AI starts to act, you know, agentically, right? So the agentic AI that trust actually begins to go down. Now we can start to hypothesize as to why some of those reasons might exist, but I think what we’re seeing at the highest level is the difference between human controlling and requesting information versus AI performing on its own, and one of the you know key there are a number of key factors, but one of the big ones that really we we see is driving down the trust is the ability to explain what the AI is doing as it’s out acting on its own. So we can get into some of that detail if you like.

Speaker 1  2:31  

Yeah, here’s the magic numbers. This report: 2,700 global IT and data decision makers across 28 countries. Pretty broad sampling, and I know you travel a lot. You might have met all those people, Reggie, through the one travel you do. So we’re talking about the phrase in the business. I think it’s post hoc review. So, so what what the study shows is that when government agencies tend to use it, they they want to reserve the final decision at the end to a human. Is that right?

Speaker 2  2:59  

That’s correct. That’s the proverbial human in the loop, as we hear it referred to sometimes.

Speaker 1  3:05  

So, I think the reason is because, at least according to the study, it’s explainability. So people say, “Okay, John, why don’t you trust this data? I’m gonna say, “Well, you know, I want to see what’s behind the curtains. I want me to. I want Reggie to show me his work on that math problem. Is that the concept here we’re talking about explainability?

Speaker 2  3:23  

Yeah, so that’s part of it. I think the other part, which goes maybe unsaid here, is rationale. Like you know, what what I’m theorizing is that you know people are what they’re calling overriding the AI, which simply means that you know you get output from the AI, and now you’re making a different choice, right? To use it or to not use it is a choice is a different choice than just by default using it, right? And so these overrides, as they’re referred to, we see start to increase as agentic as the AI becomes more agentic, and so one could argue that John that that is a function of appropriately use human judgment. Okay, right, and you know I don’t think any of us would argue that humans shouldn’t do that. Now, of course, there’s the efficiency conversation. Of course, there’s a conversation about whether humans are more accurate than AI, etc. etc. But I think we we’re at a point at which we can begin to measure those things. Right? AI makes decision. Human makes decision. Which one is more accurate, more more frequently? Which one is more appropriately accurate for a given situation, again, is a slightly different matter as it relates to judgment. And so, what the study showed was just that there are overrides. And what we have to really start to dig into more is what’s what’s driving those overrides. One of the pieces that we point out. Is explainability, but when you start to double click on explainability, what we find is that people feel like there’s a lack of context that’s being provided, and that’s what I’m referring to when I talk about that rationale, John. And so I think there’s more work to be done from a study perspective, but I think the trajectory that we see now is is pretty clear, which is as agents are beginning to act within corporations, you know, at least according to these two 700 folks, that the the level of reliability that we associate with them, or the feeling of trust, if you will, begins to diminish. Now, what I find particularly interesting is that we distinguish between get this trust, which is that feeling, and trustworthiness, which is the evidence that would support trust. And so we break this trustworthiness down into what we refer to as a trustworthiness index, and it’s got a several features associated with it. We can dig into that, but it basically they’re kind of five dimensions: data and data quality and governance is one. Model governance and oversight, explainability and fairness, responsibility, policy, and then there’s a fifth one called audit and accountability. So these are the five dimensions of the trustworthiness index, and so in that index, what you’ll see is you know mounting evidence. Now, you know the hypothesis is you know as we have this mounting evidence, then we should be able to trust more, right? But again, we’re seeing that the feeling of trust actually diminish, despite the fact that the trustworthiness index is increasing over time. So I think that’s kind of a fascinating dichotomy. That again, just a limitation of studies-you can’t study everything all at once. So I think that’s a kind of a fascinating piece that we need to explore a little bit further.

Speaker 1  7:05  

You know, Reggie, I don’t think you’re a lone wolf out there screaming from a mountain about this. I think it’s a topic that’s bubbling up. In fact, I was at a Billington cybersecurity conference last week, and one of the ambassadors for cybersecurity from Estonia, her name is Helen Pop, and her presentation. She said, “What we have here is is we have a crisis in trust, and Estonia has been a leader in all kinds of cybersecurity defenses and back and forth. So, so this trusting, it seems. I guess if you went to Google Trends and and ran a parallel search on artificial intelligence and trust, you’d find both of those little you know those up and to the right. And so this seems to be the question because my government listeners they are serious about privacy. They they are serious about compliance. They are serious about accountability, and and I think that’s good for me being a taxpayer. And they have my social security number and medical information, so it’s probably good that the governments trust AI the least, isn’t it?

Speaker 2  8:00  

So here’s where it gets really complicated, right? At some point in time, we see that we’ve got technology that can actually help us do our jobs better, right? And so, in mod with modern technology, there’s always this gap between kind of the when it’s novel, as when when it emerges, and then once it begins to go mass market, stabilize those sorts of things, right? And so I think we’re in this space of emergence right now, and what’s you know characteristic of this space of emergence is is uncertainty, right? We just lack knowledge because we lack enough observations to give us the comfort in accepting or adopting the technology, and I think this is where we are. So, I think recognizing where we are on the life cycle of this conversation becomes really important, especially as it relates to the domain in which you are adopting or thinking about adopting the technology. So, let’s break this down. If I am a small business, you know, a 10-person or less shop, and I’m trying to gain some advantage against the, you know, the the big, you know, firms in my industry, it probably makes sense for me to take some risk, right? If I am a government of 300 million people. It probably makes less sense for me to take those same risks as that small firm might. Yeah. Right. So the the challenge here is how much of the technology can I adopt as a government in order for. My government to be run more efficiently, more effectively on behalf of the citizenry, and so I think again, just recognizing the domain in which we sit and the place that we’re still at on the curve. Perhaps, and I’m not the decision maker in these things, but I do have at least one vote, perhaps working with the latest frontier models to do agentic workflows on behalf of I don’t know benefit distribution doesn’t make the most sense, right?

Speaker 1  10:36  

Yeah.

Speaker 2  10:36  

But but maybe using some of the more known capabilities, kind of more the deterministic stuff, as an example, does make sense because we we do have enough observations that would tell us that part of the technology, those sorts of capabilities, can be trusted, can be reliable, are robust enough for us to fix them should they break and those sorts of things, and so I think again, good technology leaders need to be able to identify kind of where we are on the curve.

Speaker 1  11:14  

That’s what software developers talk about their maturity level. In fact, people with discussions on zero trust talk about maturity level for adapting to zero trust. So, so you look at it as I’ve never thought of that maturity level because it’s also new, isn’t it? I mean, three years ago we wouldn’t have had this conversation, but it’s kind of just crept up on us. Maturity level, understanding safety and transparency in artificial intelligence, and big topics there, and people can avoid that, aren’t they? I mean, I would think people regarding guarding data are going to be very wary, huh?

Speaker 2  11:47  

Certainly, you’ve got a lot of wariness. I think the conversations of the day don’t help in terms of abating some of the wariness. You know, we’re having you know over the last couple of weeks, a significant AI safety conversation, effectively, and whether to pace the frontier, as they call it. You know, those are big conversations. Now, it’s also important to note here, John. Look, most of us using modern AI capabilities aren’t using the capabilities of the frontier, so to speak, and so we again most of us are using it for purposes that are perfectly adequate, sufficiently safe. Now I think as we go up the ladder in terms of risk and potential human impact, we have to be a lot more apply more a lot more rigor to ensure that you know we aren’t going on the other side of our values. But again, most of us, for the purposes of personal productivity in the workplace, are using AI for completely adequate purposes.

Speaker 1  13:06  

I’m going to read a quote from this report, and you can agree or disagree or change it or modify it. Talks about, and this is interesting. 54% of public sector organizations are still experimenting or piloting AI, but only 12% have optimized data infrastructure, and so maybe the first step is this data optimization, and the whole maybe that’s the wariness, huh? Does that seem high or low to you? I don’t know. 12% seems high.

Speaker 2  13:38  

Yeah, it could be. You know, I won’t comment on the number per se. Obviously, every one of those percentages is going to be a function of the people that you interview for your for your survey, and then you have to extrapolate. But I think again the trajectory is sound, and I think that’s what we have to look at and then measure it consistently over time. But I think the piece that you’re pointing out, as you mentioned earlier, is consistent with what we’ve seen in the past, which is that data maturity consistently holds organizations back. Right. This is, if you will, not a new phenomena as as we start to enter the agentic era. Right. Most or organizations still lack sufficient data lineage, the ability to kind of validate, you know, where their data comes from, if it’s still appropriate and valid for purpose. Most organizations still lack a sufficient amount of governance capabilities to to safeguard the data, to use it for purpose, to deprecate it as needed. Like those are those are not new topics, right? We’ve been talking about that literally for a decade plus, and. So what I think this era really starts to illuminate is, without the proper data foundation, you’ll never get to a point at which you can then optimize that data, contextualize that data, and then get great advantage from today’s AI. And so what organizations are having to do now is going retrofit as best they can, and and therein lies the challenge in terms of enterprise adoption for AI.

Speaker 1  15:34  

I listen all kinds of podcasts. I was listening to a podcast about some state leaders and a smaller state, and he said, “Well, we didn’t have any data governance policies, so he went to ChatGPT and asked Chat to recommend some data governance policies. So this is the whole whole circle here. So there are creative uses of getting around this too. So it’s like it’s not all not all doomers here. There’s there’s ways to work around it. You know, it may be a maybe a bumpy road for Reggie, but in the end, he’s he’s going to get to he’s going to get to the whatever happens to be the the mountaintop or something. So it’s it’s going to work out, but boy, it’s going to be so many bumps in the road, especially with this data quality and going. Can we trust this data? Well, can you you know can you trust the AI? Can you trust the data? This is it’s going down to this, isn’t it?

Speaker 2  16:22  

Well, yeah, you can you can run out and get the instruction from the the LLM of choice, right? Whether or not you can implement those instructions becomes a completely separate matter. Yeah, yep.

Speaker 1  16:36  

So it’s going to be upon us. Next four or five years, we’re going to have this wave of AI. We know that, and so most of the people in the study talk about increases of some big proportions next few years. So, what kind of challenges can they expect, and and what methods, what techniques, what suggestions do you have to help them overcome this this data strategy for getting reliable AI?

Speaker 2  17:00  

Yeah. So what we’re saying is that many organizations are continuing their investment in AI. And now we should be clear: some of that investment is not intentional. Some of it is purely a function of the changing economics of of using foundation models, you know, we’re seeing budget budgets balloon and and and be exceeded for for IT spend just simply because the economics have changed on on that end. But on the other end, what we’re also seeing is that organizations who and remember I talked about the trustworthy AI index, organizations that are scoring high on that index are actually seeing something like a 15x return on their AI investments. So, you know, I think it pays to invest in trust, if you will. A lot of that hinges on that that data optimization piece that I referred to earlier. But then again, we cannot overlook the the other elements around model governance and explainability and and and auditability and and so on and policies. And so I think the again the trajectory is sound, which is once organizations have those processes in place, and then they can actually habituate those processes through action. What we’re starting to see is that those are the firms that are getting the early advantage as it relates to return on AI investment,

Speaker 1  18:43  

so Reggie, this trustworthiness index, where did that originate anyway?

Speaker 2  18:47  

So that’s something we came up with with IDC last year. So last year, 2025, was the first time that we did the data and AI impact report, and what what the the big reveal there in in the 25 report was that people seemed to be responding to AI with levels of trust that exceeded what was reasonable. Put it that way. All right, and so people were responding to generative AI at a at a higher level than generative AI was demonstrating its ability to be trusted, right? So just looking at it in terms of accuracy numbers and that sort of thing. So we want to follow up on that, and what this year’s report is starting to reveal is that there is a bit of a divergence around trust, the feeling and trustworthiness, the evidence, and the the diversion begins when AI stops kind of being that research tool for us, and then starts to be be an actionable tool for us. So that’s the the delta, if you will, between. Generative and agentic AI.

Speaker 1  20:02  

So, Reggie, I think earlier you mentioned the number two was the model in the Trust Wozniak report. I mean, how can how can any models be evaluated? I mean, it’s just it seems like it’s just I give up. I mean, it’s just so many new changes and and and so many really smart people making observations that I never dreamt of anything like that. So, so it’s it’s number two out of five. It’s right behind data. Yeah. So, so how does one go about even evaluating that?

Speaker 2  20:28  

Yeah. So, that’s a great question, and I want to kind of break this down for your audience, John. So, in today’s world, we’ve got a collection of AI models, right? Yeah. When we hear the word AI model, a lot of times people intellectually drift toward the the foundation models, the big ones. You know, the Google Gemini’s, Anthropic Clouds, the OpenAI, ChatGPT, and right. That’s where people go intellectually. But the reality is, you have models that are built on those models, and then you have models that are completely distinct and separate from those models. They may not be big, you know, billion-parameter, large language models. They might be much smaller models for, say, fraud purposes, as an example. Now, that’s just kind of the setup. As we talk about model governance and oversight, when we’re talking about those big foundation models, you’ll see all sorts of evaluation methods, leaderboards, and you know, every day there’s a new announcement about this thing runs faster than that thing. This, right, and that’s that’s that’s great as it relates to the the stuff that they believe is important to measure the performance of those models. What that doesn’t do, however, John, is tell you whether any of that matters inside of the part of the business that you’re dealing with, part of the government that you’re dealing with, right?

Speaker 1  22:02  

Yeah,

Speaker 2  22:03  

and this is why I say use cases matter the most, right? Because if you’ve got a model with a an assortment of capabilities, you can use the model to I don’t know help with the case that you just suggested, which is to give me a data governance strategy. You could use that same model to give me a chicken noodle soup recipe, right? Seriously, and so the use of that model is what matters the most, and it’s the use that that dictates things like risk and reward, and so what we really need to get start to kind of create a regime around is the ability to to to monitor and assess use cases over time, and so that was a revelation that we had at SaaS some 18 months or so ago, and we actually started to build instrumentation around that idea, and so we built a product out of it called AI Navigator, SaaS AI Navigator, and so it allows us to monitor use cases as I just described, and then the use case monitoring cuts against things like policy. So, for instance, inside of every you know government agency, there are a collection of policies that need to be observed. Some of them are unique to that agency; they might be completely different in a different agency, right? Then there are policies that are external, like you know regulations and laws and such. And so, with the use of AI for a particular use case, we need to be able to monitor whether or not that particular case is drifting in or out of policy, right? And we need to be able to constrain it as such. Why that becomes a you know important is that the data underneath an AI model can adjust, and when that data adjusts, that model might drift, right? So the purpose that it was created for, let’s say, to make sure that we are extending benefits to a certain, you know, given a certain set of criteria, we want to make sure that we are including the people that are part of that criteria and excluding the people that are not part of that criteria. Well, if the data underneath the model begins to drift, then the outcome associated with those criteria may drift as well, and so just being able to see those sorts of things becomes really important. And the last and final point, and I’ll stop. We also need the ability to link out to other systems, for instance, financial systems, to determine whether or not a model is performed. Financially consistent with our expectations, and if it’s not, if we can, you know, swap that model out for a different model to perform the same use case, and so being able to kind of put that in in a single instrument, we believe is really handy for leaders, and so we, you know, there was nothing out in the market that that we saw that because we needed for ourselves that we that we saw was going to be useful for our purposes and so we said well let’s let’s take some time to build it and that’s what we did.

Speaker 1  25:32  

I got a 40,000 foot question for you and you can slam down the phone. When it comes to evaluating models, do you think the federal government should take a role in? I mean, you’ve read the newspapers, you’ve read the stories. I mean, you just gave example A of well, no, we don’t evaluate. We we the use case on it. I mean, it just seems like it’s such a a moving target. It’s almost like it’s just it’s changing too fast. And and who are they going to get to evaluate. I mean, look at SaaS. 50 years of smart people, and and and they had a hard time putting their hands around this. What are you going to do with, you know, a new government agency trying to evaluate? I I don’t. It’s a

Speaker 2  26:14  

big task. It’s a big task. But I I fall on the side of if the the one of the primary roles of our government is to protect its citizens, and it would be outside of that expectation if the government was not involved with evaluating capabilities that might harm its citizens, and so this is not something that should be taken lightly. I think we’ve got enough smart people inside of government. We have a lot of precedent already for dealing with complicated matters like this. You know, all we need to do is think about. You know, talked about cyber earlier. You talk about nuclear in the past and and the present, right? So we we have enough scenarios where there’s potential mass harm that the where the government has interceded to some degree that I have every confidence that we have the ability to do the same now. Now the one kind of distinction, and maybe you can keep me honest here. You might have some more history than I do, but a lot of kind of the the mass general purposed kinds of technologies that have been introduced to our nation over the last in my lifetime, anyway, have an origin inside of the government, and then they get introduced into the private sector after the proper guardrails are erected and that sort of thing. I think what distinguishes this moment is that a lot of the capability that we’re talking about, if not all, is private sector, right? And so the incentive structure is different. Right, right, right now there’s a lot of capital that’s chasing the AI frontier, and you know capital chases for the purpose of capital return. So the motivation is different than if it were created in the government itself. So the government being able to intercede with some sort of testing regime to evaluate models might feel like a lot more force in this scenario than it did in the prior scenarios. However, it may be a necessary force at this point in time, given the things that we can anticipate in terms of harms. Now, look, I don’t, you know, today I haven’t seen anything that would allow me to subscribe to this idea of, you know, flying robots coming and killing us all, right? But I do think that there is enough evidence to support the idea of pretty significant economic calamity, and I think that should be taken very seriously. And we should do all that we can to attempt to avoid that, primarily because economic calamity leads to social chaos, right? And those are the sorts of things that that concern me more than you know, kind of the whole existential conversation right now.

Speaker 1  29:25  

Well, that was a tough question, and thank you much for your opinion. That was very insightful. What I’ll do for our listeners’ benefit, I’ll put a link to that report in the show notes, and you can read yourself. And the whole idea of trustworthiness and trust-I think it’s a topic that’s going to be more and more on in the minds of everyone, from software developers to ethics people to to everyone, from from even even people who are worried about data on the state of Ohio’s DMV site, and they’re seriously worried about that getting out, and and what can happen because we’ve seen models that are that have holes in them, don’t we? I mean. Just because you think something’s going to happen, you don’t you don’t know what you don’t know, and and you just don’t know what’s going on. Some of these models don’t good.

Speaker 2  30:05  

Yeah, you know, back to this emergence conversation from earlier. There’s a need for an increase in digital literacy for all of us, right? We have to create it, and that’s why I like doing conversations like this, we we just need to create awareness initially, so that people can kind of get their heads in the game. That you know, everyone doesn’t need to become a data scientist or anything like that. But but when you’re dealing with cross-cutting horizontal significant capabilities like this, everyone needs to have some measure of understanding about how this stuff affects their lives, right? So if you can imagine, you know, when the internet was coming online somewhat 30 years ago or something like that, now there were a lot of people who were just on the sidelines, like I’ll never do that. But now, you know, you fast forward all these decades, it’s an indispensable utility, right? And and how do you operate in our society without some knowledge of this thing called internet? And so I think we’re headed in the same direction with AI. And so as a result, it’s really important that we start to get involved. We all kind of the big W. We we all kind of get involved in this conversation, to the degree at which we can make some informed choices about how we choose to allow AI to affect our lives.

Speaker 1  31:28  

Well, good. Thank you very much. We’re running out of time here, Reggie. You have been listening to the Federal Tech Podcast with John Gilroy. Like to thank my guest, Reggie Townsend, Vice President AI Ethics, Governance, and Social Impact at SAS. Thanks, Reggie.

Speaker 2  31:40  

You got it. Thanks, John. Appreciate it.

Transcribed by https://otter.ai

Reggie Townsend is the Vice President of the SAS Data Ethics Practice (DEP), where he spearheads a global responsible innovation initiative to develop strategies for implementing data-driven systems that promote human-centric technology practices. The DEP offers a range of services, including advisory support, product strategy and thought leadership.

In 2022, Reggie was appointed by the US Department of Commerce to the National Artificial Intelligence Advisory Committee (NAIAC). The NAIAC provides guidance to the White House and the National AI Initiative Office on various AI-related issues. He also serves on the board of EqualAI, a nonprofit dedicated to working with companies, policymakers and experts to reduce harms associated with AI.

Prior to leading the DEP, Reggie led cloud, consulting and managed services engagements for the health care and life sciences sectors. With experience in strategic planning, program management, consulting, and business development, he has a deep understanding of advanced analytics, cloud computing and AI. His extensive expertise spans multiple industries and sectors where AI is increasingly utilized, and he is passionate about equity and human empowerment. In 2021, Reggie received the SAS CEO Award of Excellence, the company’s highest employee honor, recognizing exceptional work and core values.

An inspiring speaker and presenter, Reggie has appeared on stages globally, from SXSW to Singapore, and his insights are routinely sought after by journalists around the world. He holds a bachelor’s degree in electronics engineering from Southern Illinois University and an MBA from the Illinois Institute of Technology.