Is Generative AI All It’s Hyped Up to Be?

[00:00:00] [SPEAKER_00]: Robert Half Research indicates 9 out of 10 hiring managers are having difficulty hiring. Robert Half is here to help. Our recruiting professionals utilize our proprietary AI to connect businesses with highly skilled talent. At Robert Half, we know talent. Visit roberthalf.com today.

[00:00:23] [SPEAKER_01]: Welcome to Tech News Briefing. It's Tuesday, October 22nd. I'm Zoe Thomas for The Wall Street Journal.

[00:00:29] [SPEAKER_01]: AI excitement seems to be everywhere. And while some generative artificial intelligence developments may have the potential to be transformational, others are overhyped.

[00:00:41] [SPEAKER_01]: At least that's according to Arvind Narayana, professor and director of the Center for Information Technology and Policy at Princeton University and author of the book AI Snake Oil.

[00:00:53] [SPEAKER_01]: But for business leaders looking to offer innovative products, AI could be a door to a new future.

[00:01:00] [SPEAKER_01]: The co-founder and CEO of cloud storage company Box, Aaron Levy, has said generative AI will enable Box to enter its third era as a company.

[00:01:09] [SPEAKER_01]: Narayana and Levy sat down with our tech columnist Christopher Mims at the WSJ's CIO Network Summit to discuss what generative AI can do and how it's being overhyped.

[00:01:20] [SPEAKER_01]: Today, we've got highlights from that conversation.

[00:01:29] [SPEAKER_02]: So let's start with kind of your respective positions though, because I think there is some daylight here that's worth talking about.

[00:01:35] [SPEAKER_02]: So, you know, Aaron, you and I have talked about how you are helping your customers incorporate AI into these existing workflows that they're building with your platform.

[00:01:48] [SPEAKER_02]: And you're clear-eyed about the limits of that, but you're also bullish on, oh, this is going to be transformational as we put it into various steps of these existing workflows.

[00:01:58] [SPEAKER_02]: And it's the scaffolding that you're providing.

[00:02:00] [SPEAKER_02]: So could you just describe that a little bit?

[00:02:01] [SPEAKER_02]: Because I think that differs significantly from this vision of let's just shove everything at a chatbot.

[00:02:06] [SPEAKER_04]: Yeah, actually.

[00:02:06] [SPEAKER_04]: So I think by virtue of us being an enterprise software, we can only really sell customers things that actually work.

[00:02:12] [SPEAKER_04]: Like there's literally no value in overselling something because you'll find out literally within a day that in actual runs within a customer environment, it's not real.

[00:02:20] [SPEAKER_04]: So we try and balance, obviously we want to have an ambitious vision for the future.

[00:02:24] [SPEAKER_04]: And we're getting behind generative AI because of the rate of improvement that we're seeing and how transformational it is.

[00:02:31] [SPEAKER_04]: But we have to be very clear-eyed around what use cases it can actually solve today.

[00:02:35] [SPEAKER_02]: Let's give like one specific one.

[00:02:37] [SPEAKER_02]: Oh, sure.

[00:02:37] [SPEAKER_02]: What's a basic one?

[00:02:38] [SPEAKER_04]: So AI, as I think we're all trying to deliver, is incredibly good at synthesizing unstructured data and producing insights and information around that.

[00:02:47] [SPEAKER_04]: In Box, we have about an exabyte of content, hundreds of billions of files.

[00:02:50] [SPEAKER_04]: And in every single one of those documents, whether it's a marketing asset, a financial document, an earnings release, a contract, is an unbelievable amount of intellectual property and value.

[00:02:59] [SPEAKER_04]: And right now, that data is largely untapped in most organizations.

[00:03:03] [SPEAKER_04]: So we've created a series of tools that let you basically interact with that content.

[00:03:06] [SPEAKER_04]: You can ask your documents questions.

[00:03:07] [SPEAKER_04]: You can summarize information.

[00:03:09] [SPEAKER_04]: You can synthesize it.

[00:03:11] [SPEAKER_04]: But one of the big problems that customers run into in these types of scenarios is, well, if I just put a search interface across all of my unstructured data, you're going to get lots and lots of wrong answers.

[00:03:20] [SPEAKER_04]: Because the AI is no better than a brand new human would be at figuring out what is the actual authoritative source of truth of content.

[00:03:27] [SPEAKER_04]: And so we've developed a tool called Box Hubs.

[00:03:30] [SPEAKER_04]: It lets you pre-curate and organize content that is your effective source of truth for any given topic in the enterprise.

[00:03:36] [SPEAKER_04]: But without that pre-curation, the AI is going to get a lot of answers wrong.

[00:03:39] [SPEAKER_04]: So we've had to develop a bunch of non-AI tools at the same time that we're delivering AI to be able to get the right kind of answers from data.

[00:03:46] [SPEAKER_04]: So take an invoice, a contract, a marketing asset, and be able to label it and classify it.

[00:03:50] [SPEAKER_04]: So these are very pragmatic use cases.

[00:03:52] [SPEAKER_02]: Arvin, right now we're talking specifically about this newer stuff that's utilizing so-called generative AI.

[00:03:57] [SPEAKER_02]: If we take a step back, what are the bigger issues that are unforeseen by the toolmakers like Aaron when this gets implemented at scale?

[00:04:07] [SPEAKER_02]: And we could talk about basic stuff like search and retrieval or whatever other examples you want to talk about.

[00:04:11] [SPEAKER_03]: Yeah, definitely.

[00:04:12] [SPEAKER_03]: So first of all, no real disagreement with what Aaron said there.

[00:04:14] [SPEAKER_03]: But let me also give you my kind of a little bit more skeptical perspective.

[00:04:18] [SPEAKER_03]: So we've been hearing for a couple of years that this transformation is about to happen and hasn't happened yet.

[00:04:23] [SPEAKER_03]: My position is that it probably will happen, but more on the timescale of something like a decade.

[00:04:28] [SPEAKER_03]: It's not going to be one or two years.

[00:04:30] [SPEAKER_03]: And here are some of the reasons, I think, why that is.

[00:04:32] [SPEAKER_03]: So we have general purpose AI, but its use cases are going to be very specific.

[00:04:37] [SPEAKER_03]: A lot of what people are trying to use it for is prediction.

[00:04:40] [SPEAKER_03]: To use it in automating hiring to predict which job candidate is going to be successful, for instance.

[00:04:45] [SPEAKER_03]: That's not something AI is going to be good at, no matter what kind of AI technology you use.

[00:04:50] [SPEAKER_03]: Because that's just a human problem.

[00:04:52] [SPEAKER_03]: You can't predict the future.

[00:04:53] [SPEAKER_03]: It's just inherently stochastic.

[00:04:55] [SPEAKER_03]: The universe, in a sense, doesn't know the answer to it.

[00:04:57] [SPEAKER_02]: And by the way, predictive AI was what we were all talking about until ChatGPT dropped, right?

[00:05:02] [SPEAKER_03]: That's right.

[00:05:02] [SPEAKER_03]: That's right.

[00:05:03] [SPEAKER_03]: But so even with generative AI, let me just quickly go through a whole bunch of limitations that people are finding.

[00:05:09] [SPEAKER_03]: So first of all, you have to pilot it within your organization to see if a particular use case is going to work for you.

[00:05:13] [SPEAKER_03]: Because it's just so dependent on your specifics of your data set, various other things.

[00:05:18] [SPEAKER_03]: 90% of the work is data work.

[00:05:20] [SPEAKER_03]: You have to do a lot of the evals internally.

[00:05:22] [SPEAKER_03]: And then I think this goes to what you were saying as well.

[00:05:25] [SPEAKER_03]: Because this is not like traditional software.

[00:05:28] [SPEAKER_03]: It's reliable.

[00:05:28] [SPEAKER_03]: It does the same thing over and over again.

[00:05:30] [SPEAKER_03]: It's more like a creative but unreliable intern, right?

[00:05:34] [SPEAKER_03]: And so people who are using generative AI in the workforce have to reorient their whole workflow.

[00:05:39] [SPEAKER_03]: That sort of thing doesn't happen overnight.

[00:05:41] [SPEAKER_03]: It takes years.

[00:05:42] [SPEAKER_02]: And your contention is we have to do that experimentation.

[00:05:44] [SPEAKER_02]: We don't already know what are the good use cases?

[00:05:48] [SPEAKER_03]: Correct.

[00:05:48] [SPEAKER_03]: At least what we're seeing so far with generative AI is that it's not really a technology for automation.

[00:05:54] [SPEAKER_03]: It's fundamentally different from traditional software.

[00:05:57] [SPEAKER_03]: It requires a reconfiguration of the relationship between the worker and the software.

[00:06:02] [SPEAKER_03]: We call this the capability-reliability gap.

[00:06:04] [SPEAKER_03]: If AI could do reliably all the things it's capable of today, it would truly be an economic transformation.

[00:06:10] [SPEAKER_03]: But if it's going to fail 10%, especially with AI agents who have been doing a lot of research on this,

[00:06:15] [SPEAKER_03]: if it's going to order DoorDash to the wrong address 10% of the time.

[00:06:19] [SPEAKER_03]: Those are the kinds of errors that people are actually reporting.

[00:06:21] [SPEAKER_03]: It's a completely useless product.

[00:06:23] [SPEAKER_03]: And similarly, an enterprise use case.

[00:06:24] [SPEAKER_03]: In fact, in enterprise, you can tolerate even less error, I would say.

[00:06:27] [SPEAKER_03]: So it's not a technology for automation.

[00:06:29] [SPEAKER_03]: So what is it for?

[00:06:30] [SPEAKER_03]: Maybe creativity enhancement.

[00:06:31] [SPEAKER_03]: But to take advantage of that, you need a lot of reorientation.

[00:06:33] [SPEAKER_04]: I think that what we're finding, I'm sure what you're finding is actually you probably need deterministic software at many steps along the way.

[00:06:40] [SPEAKER_04]: And you're giving sort of small bits of cognition tasks to the AI.

[00:06:45] [SPEAKER_04]: Read this document.

[00:06:46] [SPEAKER_04]: Extract this metadata field.

[00:06:48] [SPEAKER_04]: Don't then do anything with that.

[00:06:50] [SPEAKER_04]: Send it back to a deterministic workflow system.

[00:06:52] [SPEAKER_04]: And then you can begin to automate.

[00:06:53] [SPEAKER_02]: Do you still have to have a human in the loop to catch the errors that the AI makes?

[00:06:57] [SPEAKER_04]: It all depends on the task.

[00:06:59] [SPEAKER_04]: Healthcare might be one end of the spectrum where you really want a human in the loop versus something that can tolerate some degree of errors,

[00:07:05] [SPEAKER_04]: like maybe a basic customer support question where you can always escalate it back to a human if the customer is really upset with the answer.

[00:07:12] [SPEAKER_04]: We have a whole continuum of tasks to go after.

[00:07:14] [SPEAKER_04]: And I think enterprises have to figure out which tasks fall on which part of the spectrum on that.

[00:07:19] [SPEAKER_01]: Coming up, Levi and Narayanan do agree on something when it comes to adoption of generative AI, the pitfalls.

[00:07:27] [SPEAKER_01]: We'll have more after the break.

[00:07:37] [SPEAKER_00]: Robert Half Research indicates nine out of ten hiring managers are having difficulty hiring.

[00:07:43] [SPEAKER_00]: Robert Half is here to help.

[00:07:44] [SPEAKER_00]: Our recruiting professionals utilize our proprietary AI to connect businesses with highly skilled talent.

[00:07:51] [SPEAKER_00]: At Robert Half, we know talent.

[00:07:53] [SPEAKER_00]: Visit roberthalf.com today.

[00:08:00] [SPEAKER_01]: We're back with highlights from the conversation between our tech columnist Christopher Mims,

[00:08:05] [SPEAKER_01]: Box co-founder and CEO Aaron Levy,

[00:08:07] [SPEAKER_01]: and Princeton University professor Arvind Narayanan from the WSJ CIO Network Summit.

[00:08:14] [SPEAKER_02]: So Aaron, when you're talking to folks who are trying to adopt AI,

[00:08:17] [SPEAKER_02]: you know, within these systems you're helping them build,

[00:08:19] [SPEAKER_02]: what do you tell the people you're working with to help them avoid some of these pitfalls?

[00:08:24] [SPEAKER_04]: I think in the past year, in our conversations with CIOs, CTOs, heads of AI,

[00:08:29] [SPEAKER_04]: I think it's pretty pervasive, the understanding that, okay, this is a probabilistic system.

[00:08:33] [SPEAKER_04]: It's not deterministic.

[00:08:34] [SPEAKER_04]: And I think most customers are fairly pragmatic.

[00:08:37] [SPEAKER_04]: I think there's still some maybe differences in the market.

[00:08:39] [SPEAKER_04]: I'm pretty skeptical of companies doing massive sort of training runs on their own data,

[00:08:43] [SPEAKER_04]: spending just unbelievable amount of money trying to build their own model,

[00:08:46] [SPEAKER_04]: because I think the breakthroughs you're going to see from OpenAI and Google and Anthropic are just going to be,

[00:08:51] [SPEAKER_04]: and Meta are going to be, you're just not going to compete with them from a CapEx standpoint.

[00:08:55] [SPEAKER_04]: So I think you should ride the innovation curve of the hyperscalers,

[00:08:59] [SPEAKER_04]: because they're going to be doing the work for you,

[00:09:01] [SPEAKER_04]: as opposed to you trying to compete at that scale.

[00:09:03] [SPEAKER_04]: So that maybe is still something that I think the market is trying to figure out.

[00:09:06] [SPEAKER_04]: Does every company need a proprietary model,

[00:09:08] [SPEAKER_04]: or you should really care about actually your data organization,

[00:09:11] [SPEAKER_04]: and then wait for these models to get exceedingly good at working through your information.

[00:09:16] [SPEAKER_04]: So again, permissions, curation, organization, taxonomies, data structure really matters

[00:09:20] [SPEAKER_04]: if you want to get the most out of AI.

[00:09:22] [SPEAKER_03]: One quick thing I'll say to that,

[00:09:23] [SPEAKER_03]: I totally agree that building your own in-house model from scratch to compete with OpenAI

[00:09:28] [SPEAKER_03]: is maybe not the best approach.

[00:09:30] [SPEAKER_03]: But just to emphasize one thing, Aaron said, you know,

[00:09:32] [SPEAKER_03]: don't underestimate the data work, right?

[00:09:34] [SPEAKER_03]: That's often 90% of the work.

[00:09:35] [SPEAKER_03]: I think there is a lot of room for fine-tuning and customizing these models

[00:09:39] [SPEAKER_03]: for your organization.

[00:09:41] [SPEAKER_03]: And the third thing I'll say is e-vals, in my experience,

[00:09:45] [SPEAKER_03]: evaluating how AI works for your particular use case.

[00:09:48] [SPEAKER_03]: A lot of that often has to be in-house as well.

[00:09:50] [SPEAKER_03]: It can be with the help of a vendor.

[00:09:52] [SPEAKER_03]: But, you know, you need to lead that work as an organization.

[00:09:56] [SPEAKER_03]: You have to evaluate it for yourself.

[00:09:59] [SPEAKER_02]: So just one more quick question.

[00:10:01] [SPEAKER_02]: Arvind, if you were sitting on some kind of hypothetical box,

[00:10:05] [SPEAKER_02]: governance board, or advisory board,

[00:10:08] [SPEAKER_02]: what advice would you give to Aaron?

[00:10:10] [SPEAKER_02]: Like, these are the things you need to be kind of alive to

[00:10:14] [SPEAKER_02]: as you are building these systems for customers.

[00:10:17] [SPEAKER_03]: Yeah.

[00:10:17] [SPEAKER_03]: I mean, not to give Aaron advice.

[00:10:19] [SPEAKER_03]: That's the only one here.

[00:10:20] [SPEAKER_03]: But one thing we've observed over and over

[00:10:22] [SPEAKER_03]: is that diffusion of technology,

[00:10:24] [SPEAKER_03]: the way people use and adapt technology,

[00:10:26] [SPEAKER_03]: is often the speed limit to innovation.

[00:10:29] [SPEAKER_03]: Innovation can't happen in a vacuum.

[00:10:31] [SPEAKER_03]: Innovation can only happen

[00:10:32] [SPEAKER_03]: when companies look at how people are using things

[00:10:35] [SPEAKER_03]: and there's a feedback loop.

[00:10:37] [SPEAKER_03]: The development of ChatGPT was a historical anomaly

[00:10:40] [SPEAKER_03]: where it was primarily based on, you know,

[00:10:41] [SPEAKER_03]: data that's on the web.

[00:10:43] [SPEAKER_03]: I think that free lunch has come to an end.

[00:10:45] [SPEAKER_03]: Going forward, it's going to be much more

[00:10:47] [SPEAKER_03]: of this tighter feedback loop.

[00:10:48] [SPEAKER_03]: And I think Box is in a great position

[00:10:50] [SPEAKER_03]: sitting in the middle of that,

[00:10:51] [SPEAKER_03]: observing both sides of it.

[00:10:52] [SPEAKER_03]: And so you have this opportunity

[00:10:54] [SPEAKER_03]: to kind of accelerate that feedback loop

[00:10:56] [SPEAKER_03]: and capture some of the value that it generates.

[00:10:58] [SPEAKER_02]: So surprisingly, your conclusion is that

[00:11:00] [SPEAKER_02]: he should accelerate the development

[00:11:01] [SPEAKER_02]: and adoption of AI,

[00:11:03] [SPEAKER_02]: even though it's sometimes snake oil.

[00:11:05] [SPEAKER_03]: Well, I mean, focus on diffusion as a way.

[00:11:08] [SPEAKER_02]: Not the Box part, yes.

[00:11:10] [SPEAKER_03]: Yeah.

[00:11:10] [SPEAKER_03]: So here's a succinct way to put it, right?

[00:11:12] [SPEAKER_03]: People are like,

[00:11:13] [SPEAKER_03]: oh, there's so much innovation,

[00:11:14] [SPEAKER_03]: why isn't there more adoption?

[00:11:15] [SPEAKER_03]: But it's kind of the other way.

[00:11:17] [SPEAKER_03]: Adoption is the critical thing

[00:11:19] [SPEAKER_03]: that will actually enable innovation.

[00:11:20] [SPEAKER_02]: Okay, thank you both for having us.

[00:11:22] [SPEAKER_02]: Thank you so much.

[00:11:22] [SPEAKER_03]: Thank you.

[00:11:23] [SPEAKER_01]: All right, that was our tech columnist,

[00:11:24] [SPEAKER_01]: Christopher Mims,

[00:11:25] [SPEAKER_01]: speaking with Box CEO

[00:11:27] [SPEAKER_01]: and co-founder Aaron Levy

[00:11:29] [SPEAKER_01]: and Arvind Narayanan,

[00:11:31] [SPEAKER_01]: professor and director

[00:11:32] [SPEAKER_01]: of the Center for Information Technology Policy

[00:11:35] [SPEAKER_01]: at Princeton University.

[00:11:37] [SPEAKER_01]: And that's it for Tech News Briefing.

[00:11:39] [SPEAKER_01]: Today's show was produced by Julie Chang

[00:11:41] [SPEAKER_01]: and Pierre Bien-Aimé

[00:11:42] [SPEAKER_01]: with supervising producer Catherine Millsop.

[00:11:45] [SPEAKER_01]: I'm Zoe Thomas for The Wall Street Journal.

[00:11:47] [SPEAKER_01]: We'll be back this afternoon

[00:11:48] [SPEAKER_01]: with TNB Tech Minute.

[00:11:50] [SPEAKER_01]: Thanks for listening.

[00:11:51] [SPEAKER_01]: Thanks for listening.