Conserving tempo with AI startups “working round with scissors” — to make use of Steve Santana’s metaphor — could be unnerving for established enterprises. However the CIO of Princeton-based Schooling Testing Service (ETS) believes corporations can’t stand nonetheless.
On this installment of the IT Leaders Quick-5 — InformationWeek’s column for IT professionals to realize peer insights — Santana acknowledges that AI presents main safety and knowledge administration considerations for enterprises. Nonetheless, he stated he’s a agency believer that when used inside correct guardrails on a basis of buyer belief, AI might be crucial to scaling ETS and enhancing buyer expertise.
He additionally forecasts that AI will essentially reshape how enterprises method cybersecurity and the scale of their safety groups.
ETS gives educational evaluation checks, such because the GRE. The group conducts 50 million educational checks a 12 months, and Santana’s objective is to make use of AI to scale check distribution to exceed 100 million checks a 12 months.
Previous to his position at ETS, Santana was senior vp and CTO of Pearson, held director-level positions at Ciena and Nortel, and was a supervisor at Apple.
This column has been edited for readability and house.
Steve Santana, CIO and head of AI, ETS
The Choice That Mattered
What choice — technical or organizational — made the most important distinction just lately, and why?
For us, it is the speedy adoption of AI. We’ve got been utilizing machine studying fashions for 20 years. All of our core scoring engines, the e-rater and SpeechRater functions, we constructed primarily based off our knowledge. They rating simply in addition to the people and are very dependable, however they’re costly to take care of since you’re consistently updating these fashions and cleansing the information.
Giant language fashions are a lot cheaper to combine with, so we wish to transfer in that course — not simply on scoring, but additionally on merchandise era, that are the check questions.
A big language mannequin can actually enable you in working the check by taking the test-taker by way of a course of. As an alternative of simply multiple-choice questions, I can begin having a dialogue with you, after which I can have brokers inside that mannequin work together with you within the ways in which anyone would in a real-world scenario. We are able to course of that you just perceive the method, the method, the ideas, the competencies behind it.
We needed to make use of this know-how, and we noticed a fantastic alternative — but additionally an incredible quantity of threat . Like at each firm, there have been plenty of their safety points. We do not know the place this knowledge got here from. How can we handle and work with it?
I in a short time simply stated that we do not have a alternative — we now have to do that. There’s so many startups which might be simply working round with scissors, that means they are going to do that with out us, and they’ll begin breaking belief with folks. We must always undertake it, and we must always present folks the best method to go about doing it.
We take the 78 years of experience that ETS has and the many years of information, and use that as a mechanism to leverage that belief. We already had excessive belief in our assessments. How can we construct it in a method that folks will belief what we do with AI? We do not use any knowledge that comes from the massive language fashions. It’s all the time utilizing reference knowledge that’s first-party knowledge that we perceive and make the most of.
A couple of 12 months and a half in the past, we determined to centralize this effort again underneath the know-how group. We’ve got a seven-point plan of how we’re implementing this know-how, and lots of issues are literally already in manufacturing at present.
We took the learnings of what everybody was doing by way of experimentation, after which we thought-about how we implement it at scale. A very powerful factor that we had was belief — our prospects trusted us. We have to preserve that as we deploy AI in these new checks and fashions, and they’ll assist us drastically cut back prices.
The Arduous-Gained Lesson
What didn’t go as deliberate just lately — and what did it pressure you to rethink?
A couple of years in the past, at a special firm, we have been pushing the enterprise to deploy a brand new CRM that was core to their enterprise. We pushed them fairly exhausting on making the choice to maneuver ahead as a result of they wasted hundreds of thousands of {dollars} on failed initiatives as a result of they did not transfer [quickly enough].
However, we had some decent-sized issues with the CRM once we rolled it out. Most of them have been issues that have been missed — enterprise necessities that we did not collect. If we had taken some extra time and had not pushed the enterprise so exhausting on that, I believe it will have gone smoother. After I pushed them, they weren’t on my facet. And when issues went fallacious, they have been nonetheless not on my facet.
Transferring ahead, I spend much more time with the enterprise group, and I attempt to be sure that we’re on the identical facet. I used to be focusing an excessive amount of on the necessities of the know-how and never sufficient time specializing in the enterprise change administration facet of issues.
The Expertise Commerce-Off
The place are you investing in expertise proper now — and what are you consciously not investing in?
The utility participant is much more vital than they was. Lots of people are very centered on being sturdy in a selected area. A utility participant is somebody who is aware of a little bit about plenty of issues and may use agentic AI to spherical out the remainder of their supply capabilities.
If in case you have a really sturdy working information of a lot of domains, AI may also help provide you with depth in these areas that perhaps you did not have entry to earlier than. You will have engineers that wish to do enterprise analyst work.
If you happen to’re an engineer and you could convert enterprise necessities into consumer tales, then you may have the AI try this. AI may also help pull all of that collectively for you and write your tales. You’ve got simply now expanded your skill to ship extra and your worth to the group has exponentially grown.
I am not investing in one-dimensional participant database directors [DBAs] that simply know every part about database efficiency administration — these issues are going away. We have inspired our DBAs at ETS to study extra items of the enterprise and construct out their [roles as] utility participant throughout these issues. However we do not hunt down particular particular person area information.
The Exterior Sign
What current exterior growth is most probably to vary how your group operates, even not directly?
The simple reply is agentic AI and the way it will influence our safety. Agentic AI can actually assist determine vulnerabilities and enhance monitoring, nevertheless it additionally massively will increase the danger profile. We’re a decent-sized group, so we now have an enormous safety group, however there are a lot of small and medium companies that do not have that safety administration.
That retains me up at evening — I used to be frightened earlier than Anthropic Mythos got here out. I am glad that Anthropic is offering organizations with the tooling to attempt to do away with their vulnerabilities. However when you have software program that is greater than 5 years previous, you most likely nonetheless have numerous vulnerabilities that you just’re having to handle. I fear about organizations that simply cannot scale as much as deal with that.
Nevertheless it’s additionally an enormous alternative. I believe safety operation facilities are going to be a fraction of the scale they’re at present. In the following two or three years, you are going to see plenty of sort of altering of these [security] assets, as a result of the AI can do the monitoring and studying to take motion on issues a lot quicker than folks can.
The Perspective Shift
What have you ever learn, watched or listened to just lately that modified how you concentrate on management or know-how — even barely?
I watch plenty of YouTube channels and ship issues off to my workers a couple of new Microsoft function or a frontier AI beta launch. It is cool spending a little bit time to be aware of how corporations are deploying their know-how and the way we are able to deploy this to our prospects.
I additionally just like the podcast Masters of Scale, which talks about how startups scale their corporations. You find out how their enterprise manages development and what they’re prioritizing.
I additionally just like the YouTube channel Ship it Weekly. It is in regards to the lowest-level know-how change replace — like other ways to configure and handle GitHub flows. It provides me little nuggets of data, so once I’m in an operations evaluation or speaking to builders, I sound extra knowledgeable, and I can ask extra questions. The builders typically begin to consider issues a little bit otherwise as a result of they know their leaders are being attentive to these particulars. That offers us a extra dependable product on the again finish.
Get extra IT management updates and insights 3 times every week direct to your inbox with the InformationWeek publication.
