Maximizing ROI: The Intersection of Cloud Cost Management and AI Innovations in Business with Erik Peterson

[00:00:02] Not just another AI conversation, I promise. Let's talk about the cost behind it. How do you optimize and understand the cost of the use of these models and how do you measure it? Can you link it to ROI? Erik Peterson is the Founder and Chief Technology Officer of Cloud Zero joins me to talk about that and his approach on this bonus episode of The Business of Tech.

[00:00:25] This episode is supported by Synchro. Synchro, the integrated remote monitoring and management and professional services automation platform, is designed for mid-sized and growing managed service providers. Its latest innovations include an AI-powered smart ticket management system with automatic ticket classifications, guided resolution steps using pre-approved scripts, and a natural language smart search function. These tools streamline ticket handling and improve response times. Discover more on the web page.

[00:00:55] That's synchromsp.com. Erik, well, welcome to the show. Erik Peterson, Hey, Dave. Thanks for having me. It's great to be here. Erik Peterson, So let's dive right in because Cloud Zero has been focused on cloud cost viability, visibility, and optimization, solutions and strategies with a platform. That's a space that has made a ton of sense for a while now, particularly as we moved into cloud and then people didn't understand how they were spending their money, realized there was a lot of money.

[00:01:25] A lot of waste in there and started contracting it. I think that's a space that most of us are starting to understand and understand where the value is. What you've done recently is you've layered on top of that AI cost optimization. Okay, I'm intrigued because this feels like a really messy space when in some levels, I don't even know the cost going in at all what it should be based on inputs and outputs. Talk to me a little bit about how you've approached this problem and how you're addressing it.

[00:01:53] Erik Peterson, So there's an interesting word you just used today, viability, which I think is a good kind of way to step back and take a different perspective on this. Erik Peterson, Because a lot of us coming from maybe traditional IT or data center worlds have seen this as cost, right? In the cloud world, we need to shift our perspective to see it as an investment and we should expect a return on that investment. And if we don't, then we don't have a viable investment. We don't have a necessarily a viable business even.

[00:02:22] Erik Peterson, Many of the digital businesses that folks are building, the machinery, the manufacturing line, the digital products that they produce, they come out of this cloud environment, but they're not thinking about it as an investment. They're not thinking about the return on that investment.

[00:02:37] Erik Peterson, Because in the reality is every dollar I spend in the cloud, whether it's AI or building a traditional website, I should be seeing a return on that. And ultimately, if you're understanding your costs, not in terms of how much you're spending, but more in terms of your margins or what you're getting in return for that, you're going to have a better outcome overall.

[00:03:00] Erik Peterson, So that's what Cloud Zero is focused on the unit economics, the deeper understanding of how you're getting a return on that investment. And that's been my focus for well, well over a decade now, even before I started Cloud Zero. And it's changing how people think about the money they're spending on cloud.

[00:03:17] Erik Peterson, Now, bring us here to this AI world. In the AI space, this question of what does it cost to do the thing that it is that I do is now critical. Companies are running all these experiments, they're building chatbots or something a little bit more interesting than that.

[00:03:32] Erik Peterson, And they are not understanding the cost of that prompt or that inference or that conversation that they're having on the website. And they're being wildly successful, but they might actually have a success failure because they're spending money and not seeing a return on that investment. Okay, let's start with the philosophy of the way you approach measuring ROI on cloud broadly, because it feels like you have to understand that to get into how you do it with AI. So walk me through that, how you understand that.

[00:04:02] Erik Peterson, So we start by first understanding what it is that your business does. It doesn't really help you to understand your costs unless you understand how to connect the business goals with the technology decisions that you're making. So we link those two things together in our product. We essentially have built a way to program our solution so that when you access our platform, you're seeing your spend directly as it relates to your business.

[00:04:26] Let's say your DoorDash. You're interested in what it costs to deliver a meal or your Peloton. What does it cost when somebody gets on that bike, right? Those are costs related to directly to what you're building, delivering, serving your market. And that is a way more useful, interesting and impactful metric to know than how much money I spend on EC2. Nobody really cares about that.

[00:04:50] But if I'm in the business to deliver, let's say a widget and it costs me via the cloud $5 to deliver that widget, but my sales team is only selling that widget for $1. The sooner I can understand that, the faster I can get to prioritizing what my services teams are doing, my engineering teams, how I'm building my software.

[00:05:11] So that's our approach to this. That's our general approach to this, where the cloud providers and the folks out there today are very much focused on telling you what you're spending on infrastructure. What CloudZero is focused on is telling you and giving you insight into what it costs to run your digital business, essentially. Now, what does AI layer on top of that in terms of complexity? Because it feels like there should be a lot of complexity there.

[00:05:35] Oh, there's an amazing amount of complexity. It starts with the one, the just stratospheric kind of costs associated with AI right now, particularly if you're trying to get your hands around some of the more advanced GPU computes. That's a constrained resource. That's a constrained resource. People are fighting over that resource. There's a real complicated kind of equation in understanding how the utilization of that or the reservation of that compute resources available.

[00:06:02] Now we're starting to see even more complicated things layer on top of that. Right now, the rage is talking about RAG or agents or how I layer these things together. And a lot of activities get kind of put underway when you fire up some of these systems where you have LLMs talking to LMs and all kinds of data flowing throughout your system.

[00:06:26] So what we see is this kind of massive halo effect where you go, oh, well, I'm doing this experiment or I'm building this particular AI component. But now it's moving a terabyte of data across the network. Now it's firing up a whole bunch of compute behind the scenes. Now it's driving new systems that kind of come online that you're not even aware of because you're using a lot of managed services that are provided by your cloud provider or partners that you're working with.

[00:06:53] And it's really hard to glue all that together so you can get to that answer. What does it cost to do that thing that it is that we say we're doing here? That's the hard part that is getting only more complicated with AI. Now, AI feels like an area where this can get really dangerous really fast. Notoriously, OpenAI isn't making any money. In fact, it costs them more to run the model than they're making in on subscription revenue. If the reporting, I mean, I'll look at the reporting and say that's the way it looks to me.

[00:07:23] It feels like you would have really unique insights on the cost models of these models for customers. How much of this is a problem for customers where they're spending more than they're getting back in ROI? And what directionally are you seeing happening here? Yeah, this is a huge problem. And the other part of the problem is that it's different in many ways depending upon what you're trying to do. First, not everybody in the world is training their own models.

[00:07:51] In fact, I think a lot of people started out thinking, okay, I'm going to have to go get a bunch of stuff, compute, going to go train my own model, going to feed a lot of data in there. And this is the most expensive way, right? We can take a look at what it costs to train the latest version of ChatGPT. It's well over $100 million. Like, there's very rare error. You're not necessarily in that category of training your own model.

[00:08:15] That said, these foundational frontier models that are now getting easier and easier to put to work within the cloud providers themselves, you know, they're creating frameworks to make this easy. There are a lot of choices. And you can see the constant kind of evolution here where we see, oh, well, Claude, I'm going to use Claude 3.5. Oh, 3.7 just came out. Oh, now we're all talking about DeepSeq. How can I get DeepSeq into here? Is that going to be cheaper?

[00:08:42] And you might have experiments happening all over your organization, and you have no way of really understanding who's doing what or the quality, both in performance and cost, that you're getting out of all these choices. So that's one of the things that is really important to understand. When you make changes, when you add improvements, because the space is moving so fast, what are the ultimate kind of ramifications of those decisions that you're making? And that's an enormous amount of data to bring in and correlate.

[00:09:10] And it goes beyond just what it costs, but how all these things work together. Now, instinctually, I would think you're not only measuring the technologies, but also then using them to be effective in doing this. What have you learned about the use of it from your own experiences on cost optimization? Yeah. Yeah. So there is a lot of very interesting things that start to happen when you try to go beyond the simple prompts or conversation that you're having with ChatGPT.

[00:09:39] You know, hey, tell me, you know, give me some dad jokes for my kids tonight. To I'm going to take a terabyte of cloud cost and usage data and pump that into a system, and hopefully it can pump out some insights that I wouldn't figure out otherwise. You have context windows that you have to deal with. You have kind of complex nuances of the data.

[00:09:59] And so this is where these ideas around agentic workflows and building agents that can be very specialized in answering questions are really taking hold, because now I can decompose this problem. So I might build an agent that understands deeply all the nuances of cost and usage within AWS and within GCP and within Azure. And then I have an agent that understands all the nuances of what you spent last week or last month within the system.

[00:10:28] And these two agents might have a conversation with each other from a conversation that I'm having with another LLM where I'm asking the question, hey, what did I spend last week? And if I wanted to spend that money somewhere else, is there a place where I could be more efficient about it? Now you set off this workflow of a conversation behind the scenes where literally the computers are talking to one another.

[00:10:49] And then you get an answer back that is actually typically more accurate because you have compartmentalized or isolated these components. And you avoid some of the earlier challenges that we saw, hallucinations, you know, the LLM just flat out lying to you because at each of those steps, you can also validate that. And that's what I was talking about earlier, where this kind of like these workflows, these agentic workflows, they're getting very complex.

[00:11:16] I think we're it's almost like the early days of Kubernetes where people were just starting to build these containerized systems. And now we're in this world of like massive microservices organizations, the huge Death Star graph of things talking to one another. The same thing is going to happen with multiple LMs, multiple agents talking and communicating and spending money all over your organization. Yeah, well, we're talking the latter half of Q1 2025, right?

[00:11:45] So how much of this is real? Because when I when I poke at agentic, the consumer side of this is a lot of hype and not a lot of reality. I don't think open AIs operators particularly impressive project. None of it really works. And Salesforce coming off the fact that agent forces and being deployed. Yet at the same time, I keep hearing about adoption of agentic AI and enterprise. I would think you've got a pretty good sense of looking at this if you're trying to measure all the cost of it. How much of this is real right now?

[00:12:14] Yeah, the the things where I see people having the most success and that's the most real in that sense is going after these very specific, specialized kind of problems where you can validate the outputs that are coming from these systems. Right. You know, a lot of the consumer work right now is happening around these very generalized, very kind of blue sky. You know, anything goes kind of kind of problem set. And that is the hardest of anything to solve. Right.

[00:12:43] When we get kind of deeper into the enterprise where I'm seeing the organizations that in particular that we're working with, they've said, you know what, this part of the business or this part of our tech stack to answer this one question or do this analysis for a customer. It's really complex and we really have a hard time understanding it. But we know how to validate the output of that.

[00:13:06] But they're finding that they can have a huge success in like applying Gen AI to that that problem because they can ask it the question. They've trained and kind of pre loaded a very complex prompt into that into that system. And then when they get the answer back, they're putting it through a validation step that if it fails, feeds it back in and keeps going. And it's kind of a recursive model here until you get the answer that that passes.

[00:13:33] That produces a really high quality output that enterprise customers actually can trust. You know, so if I'm, you know, take Cloud Zero's platform. If someone asks, how much money did I spend yesterday? I cannot have the system lying to them. It has to be accurate. We're talking about financial data here. So it's very important that we focus in on those use cases that can be validated. And that's where I'm seeing a lot of power. So I think that's what will start to happen is the stacking of these workflows. That's what we mean when we say agentic workflows.

[00:14:03] It's like we've got lots of agents, specialized, really good, accurate. Now we're combining them together to create these higher order systems. But it's not solving a generalized problem. It's solving these very specific problems that we've had for a long time in whatever systems we were building. But doing it in a way that's ultimately more cost effective, more flexible. And I can go get my teams building more innovative systems on top of that. So I want to make sure I leave our listeners with a real actionable piece.

[00:14:32] So my last question really here is to give me a sense. You've probably given a lot of thought of the balance of focusing too much on cost details with cost management versus innovation. Because you can focus too much on cost, which drives down innovation. Talk to me about how you approach that balance to make sure that cost efficiency doesn't destroy innovation.

[00:14:52] Yeah, it's easy to think that a focus on cost is going to slow down your engineering efforts or take your teams away from building more innovative systems. But the evidence just isn't there. We've heard folks started to talk about Jevon's paradox, for example, which is that the better you understand something, the more optimized and efficient you make it, the more usage you actually drive. And this is precisely what we see.

[00:15:20] But more importantly, what we see is a focus on cost as a non-functional requirement or a component of how you build and think about designing and creating systems actually drives innovation and pushes teams to do things that they wouldn't ordinarily be capable of doing in a shorter period of time than they would have been ordinarily capable of doing. And the most interesting example, I think, is, of course, what we saw with DeepSeek.

[00:15:47] Here is a team working on trying to build a more effective LLM in the market. And there's a lot of debate about exactly how much all the R&D costs leading up to that before they got to the point of the final training model, which they say ran somewhere between $5 and $6 million. But at the same time, that was all driven by cost. That innovation was driven by cost.

[00:16:12] And that is the only constraint, particularly when we're operating in a cloud environment, that exists anymore. I kind of have unlimited scale, but I don't have unlimited wallet. And by using that, I can push my teams to adopt new technology, to innovate faster, to build new functionality, and ultimately create products that lead markets.

[00:16:34] And I'm seeing it all over the place with our customers who have moved beyond just trying to find the coins in the couch to using cost as an innovation driver. And I would encourage any company to think about that from the start, not after they have a problem and they have to come to a full stop and try to understand their cloud build. Doing more with less is a great way to leave this. Eric Peterson is the founder and chief technology officer of CloudZero, a company specializing in cloud cost optimization.

[00:17:02] With over two decades of experience in software startups, Eric has been instrumental in pioneering engineering-led cost optimization and unit economics. He's an active contributor to the FinOps and serverless communities, frequently sharing insights on cloud economics, DevOps, and security. Eric, this has been fascinating. Thanks for joining me today. My pleasure. Thanks, Dave. The Business of Tech is written and produced by me, Dave Sobel, under ethics guidelines posted at businessof.tech.

[00:17:29] If you've enjoyed the show, make sure you've subscribed or followed on your favorite platform. It's free and helps directly. Give us a review, too. If you want to support the show, visit patreon.com slash msp radio, and you'll get access to content early. Or buy our Why Do We Care merch at businessof.tech. Have a question you want answered?

[00:17:52] We take listener questions, send them in, ideally as a voice memo or video to question at mspradio.com. I answer listener questions live on our Wednesday live show on YouTube and LinkedIn. If you've got a comment or a thought on a story, put it in the comments if you're on YouTube, or reach out on LinkedIn if you're listening to the podcast. And if you want to advertise on the show, visit mspradio.com slash engage.

[00:18:18] Once again, thanks for listening, and I will talk to you again on our next episode. Part of the MSP Radio Network.