10.24.2025

ForwardThinking 🤔

Let’s play the blame game

Blame it on Cain
Don't blame it on me
Oh, oh, oh, it's nobody's fault
But we need somebody to burn
~Elvis Costello, Blame it on Cain

You might have heard that large swaths of the internet went down on Monday. The culprit was a defect in the DNS automation system affecting DynamoDB endpoints at Amazon’s US‑EAST‑1 data center. Because DNS translates web addresses into server locations, this failure prevented software from finding the addresses of AWS’s database servers used by thousands of apps. If DNS can’t do its job…well… as we learned, sites go crashing down. It wasn’t quite as bad as CrowdStrike’s faulty update last year that caused worldwide IT chaos, but it was pretty terrible nonetheless.

Bad enough that I reckon that AWS SREs (Site Reliability Engineers), who are charged with keeping the system up, while scrambling to resolve the problem, were playing the blame game, at least in their minds. Andy Jassy and the rest of the C-Suite must have been seething. An outage like this is embarrassing. An outage that lasted almost 15 brutal hours is devastating.

It affected airlines — and might have been the cause of my extensive travel delays on Monday that resulted in me canceling my trip — along with sites like Snapchat, Reddit, Zoom and Disney+, as well as banks and a whole host of other sites we probably never even heard about including a slew of startups. Through a quick informal survey of boldstart’s portfolio companies, I learned that anyone on AWS was impacted at least to some extent.

So who’s to blame here? When systems fail, there is usually a post-mortem, and AWS has already explained the root cause and their prevention plan. Instead of pointing fingers, learning and understanding and trying to prevent similar incidents is probably the best way forward for both customers and Amazon.

Stuff’s gonna happen

As any IT pro can tell you, when you’re running a complex system, stuff happens, and probably more often than you think. The difference is that these days, instead of bringing down your company’s email server, it’s bringing down a good percentage of the internet. It shows the interconnectedness of today’s systems, and the nature of dependence on centralized cloud vendors.

Back in 2017, AWS had another outage at US‑EAST‑1. This one involved S3 storage, and like this week’s incident, it caused a number of sites to go down. The difference was that outage was resolved in four hours instead of fifteen —a much faster outcome, even though in the moment it felt like a big deal.

AWS Virginia data center - Photo courtesy of AWS

In a TechCrunch article I wrote at the time, Ben Kepes, a cloud computing analyst and commentator, explained that it was part and parcel of buying a complex set of services. These types of incidents inevitably happen, and what he said 8 years ago is just as applicable today.

“If anything, the outage showed just how many third parties rely on AWS for their infrastructure. The reality, as unpalatable as it sounds, is that failures happen from time to time and organizations need to plan for that failure,” he said.

Preventing future incidents instead of placing blame

AWS took a number of steps to prevent this from happening again, which you can read about in the company blog post if you want to get down deep in the technical weeds, but suffice to say, they are trying to build protections into DNS automation system to keep this from happening again.

People who deal with disasters for a living don’t find playing the blame game particularly useful. They’d rather discover the causes and take actions that can, to the extent possible, prevent lightning from striking twice. Nora Jones, founder of Jeli (an incident management startup acquired by PagerDuty in 2023, which was part of the boldstart porfolio) shares this philosophy. In a 2022 TechCrunch interview, Jones said Jeli’s approach isn’t about assigning blame or firing people, which she sees as counterproductive: “Instead of asking ‘Who did this?’ our platform asks, ‘How was this possible?’ Because if one person could make that mistake, another likely will too,” Jones explained.

Wooden Scrabble tiles arranged on a white surface spelling the phrase “LEARN FROM FAILURE.”

Photo by Brett Jordan on Unsplash

But there's another side to this story regarding what customers can realistically do to blunt the impact of an outage like this. There were sites that stayed up during the incident. The question is what did they do differently? Adrian Cockcroft, an IT veteran and former AWS employee, says the difference was that those companies had a working fail-over plan. That means they had set up a way for the service to fail over to another Amazon region as soon as their primary one ran into trouble.

“It’s possible to keep running when a cloud region goes down. As far as I can tell Netflix and Capital One stayed up through the outage(s) today,” Cockcroft wrote in a LinkedIn post.

One thing we can be sure of is that no matter how well we plan, there will always be a unique set of circumstances, or even just plain human error that can bring systems down. When that system happens to run a good portion of the internet, it makes it all the worse. But we can look for somebody to burn, or can figure out what happened and try to prevent it from happening again. That seems like a more reasonable course of action to me.

~Ron

What's new on the blog 📰

Samsara is using AI to steer better driving behavior

I spoke to Samsara VP of product, engineering, AI and safety to learn how his company combines in-vehicle cameras with data analysis and AI to help encourage fleet driving safety.

One thing is clear, he is enthusiastic as any executive I have spoken to when it comes to working with startups to improve his industry and enhance his company's offerings.

“If there is one thing I want to campaign for, it’s that this industry needs more startups. Please come start companies in our industry. It’s great when new players come in and push the thinking," Land said.

Read full article>>

Crusoe is building data centers where power is abundant and cheap

Today's AI data centers require a ton of energy to run. That's why Crusoe, a seven-year-old startup, decided to build in Abiline, Texas of all places.

It's actually one of the windiest places in the country and it had an abundance of unused wind energy capacity.

“AI needs a lot of energy. They have a lot of energy, and it made sense for us, instead of trying to build the next data center in Northern Virginia, to focus the demand for compute to areas where we can access low cost energy," Crusoe CEO Chase Lochmiller said.

Read full article>>

Cars driving down a highway toward a large wind farm, with dozens of tall wind turbines spread across open farmland under a cloudy sky.

Photo by Ron Miller

How Liberty Mutual’s CIO is leading the shift from AI experimentation to execution

As global CIO at Liberty Mutual, Monica Caldas is leading her company through the major shift that AI is bringing to all organizations. I spoke to her recently about her approach and how she balances modernization and governance inside a large organization.

As she examines individual use cases for AI, she's looking at how they can rethink workflows and processes.

"It's not just about choosing use cases. It's really about thinking broadly about processes and figuring out how to rewire parts of them.”

Read full article>>

Amazon and Google look to hook enterprise AI buyers with new tools

With OpenAI sending out a continuing onslaught of news, the hyperscalers like Google and Amazon may want to remind enterprise buyers they aren't going anywhere, and by the way, they may have a better enterprise value proposition than OpenAI.

Read full article>>

A group of bright orange goldfish swim together in clear blue water toward a fishing hook baited with a worm.

Photo by Igor Omilaev for Unsplash+

Becoming an AI-first business requires a total organizational mindshift

I recently had the pleasure of interviewing Brian Solis, global head of innovation at ServiceNow and author of the book Mindshift.

We talked about how becoming AI-first is not just a technology problem. It requires a new way of thinking about how you run your business, an effort that is often led by visionary employees, who may not necessarily come from the C-Suite.

“AI cannot work in the box of business as usual. It requires actual transformation, mostly from a data standpoint, which then involves breaking down silos and rethinking operational structures," Solis said.

Read full article>>

As OpenAI becomes the 10,000-pound gorilla, where do startups fit?

As OpenAI continues to grow and add powerful new features, it raises a question of where startups might fit in that picture. Speaking at the Madrona IA Summit 2025 , OpenAI chief strategy officer Jason Kwon offered some advice and I took note.

Read full article>>

Purple OpenAI logos floating on a purple and black striped background.

Photo by Mariia Shalabaieva on Unsplash

News of the Week 📣

Adobe FireFly foundry lets companies tune models for their brand assets

Every company has a set of brand assets that defines the look and feel of the company and who they are to the public. Iconic brands are instantly recognizable. This week, Adobe announced the Firefly Foundry, which enables brands to tune their models to that unique set of brand assets that defines the company.

Adobe has long been the center of the creative process, dating back to the 1990s when it released Photoshop. "The base models don't understand brand or franchise, so if you try to prompt it for a particular brand, if it doesn't understand that IP," Hannah Elsakr, vice president of Generative AI & new business ventures at Adobe told FastForward.

Photo courtesy of Adobe

After training, the resulting AI model is customized to the company's IP, yet kept private for the customer, who benefits from the knowledge built into Adobe’s Firefly models, supporting formats like image, video, vector, audio and 3D.

The product really takes advantage of all the Adobe creative tools, along with the generic Firefly models to give companies a model that's tuned to their specific requirements. "So they end up with a private, bespoke IP-protected for your enterprise-only model that supports their marketing use cases, creative use cases, and can be integrated into the entire Adobe ecosystem," she said.

The company announced it this week, but is planning a coming-out party next week at the Adobe Max Conference.

Openai enters new browser wars with release of chatgpt atlas

Those of us of certain age remember the old browser wars. Back in the 1990s at the dawn of the World Wide Web, the two main players were Netscape and Microsoft Internet Explorer. The clear winner of that was Microsoft, which took advantage of its Windows market power dominance to control the market until Chrome came along in 2008 and overtook Internet Explorer by 2010.

Now, we have a new browser war around AI models, and this week OpenAI entered the fray when it released its entry ChatGPT Atlas. The browser is built on the open source Chromium browser, so it's going to look a lot like Chrome with some OpenAI frosting on top.

OpenAI Atlas browser window displaying a new tab where a user has typed “vrbo.com” into the address bar, with autocomplete suggestions appearing below, including the main Vrbo website and login options.

Atlas Browser - Photo courtesy of OpenAI

The company claims the primary advantage of having ChatGPT built into the browser experience is not having to change context to answer questions. "With Atlas, ChatGPT can come with you anywhere across the web—helping you in the window right where you are, understanding what you’re trying to do, and completing tasks for you, all without copying and pasting or leaving the page," the company wrote in a blog post.

I've found this to be the case with Comet, Perplexity's browser, which I've been using mostly for research purposes for the last several months. Google is also building AI into Chrome. It's interesting to note that the DOJ had proposed forcing Google to sell Chrome as part of the remedies phase of its antitrust case last summer, but the judge involved ultimately let the company off the hook in the final ruling.

Regardless, Atlas is only for Mac users for now, but the company reports that Windows, iOS and Android are coming soon.

Anthropic and google announce latest multi-billion dollar Ai-related dalliance

It seems a week can't pass without one model or another making a huge deal with a cloud vendor in the never-ending thirst for compute capacity. Usually it involves OpenAI, but this week Anthropic got its moment in the spotlight announcing an agreement with Google Cloud with the potential to reach tens of billions in value.

The deal involves access to up to one million Google TPU chips, which should give the startup plenty of juice as it continues to compete hard in the large language model market. That kind of capacity adds up pretty quickly. "The expansion is worth tens of billions of dollars and is expected to bring well over a gigawatt of capacity online in 2026," the company wrote in a blog post announcing the deal.

Photo by Ron Miller

The company has taken a diversified approach to chip usage with deals to use Google’s TPUs, Amazon’s Trainium and NVIDIA’s GPUs. "This multi-platform approach ensures we can continue advancing Claude's capabilities while maintaining strong partnerships across the industry," the company wrote.

Anthropic made sure to articulate its commitment to Amazon in the announcement, which has been a strong partner, having committed up to $8 billion to the company. As with all these deals, it's important to note that while the numbers are big, they are dependent on completing the full extent of the arrangement. You will note the careful language that they plan to access "up to" a million TPUs. There are probably out-clauses aplenty, just as with the big OpenAI deals, but it's a good deal on paper for both companies.

My adventures in AI 🤖

Photo by Andy Kelly on Unsplash

This week, I tested Mistral for the first time. Since it was my initial foray with the model, I gave it a pretty easy test to fact check the article I had just written, a task I had already given ChatGPT, which missed the fact that I got the location of the data center wrong.

Mistral impressed me by passing the test with flying colors, finding each claim, checking it against sources and confirming that my information was correct (or mostly correct anyway).

I think I'll be adding it to the mix, especially as I continue to be disenchanted with Gemini's fact checking.

I also had it do an edit, and while the jury is still out on that one, I'll continue to mix it in and see how it performs. One thing I find annoying is it opens up a new browser window to chat with it, which is pretty annoying for my use case of generating a table of edits, then reviewing the table and implementing the ones I agree with in the document. It's just harder to switch between browser windows than it is tabs.

This post originally appeared on my LinkedIn.

What I'm reading 📚

Person reading book, but all you can see is their hands holding the book. They are sitting on a hardwood floor steeped in shadow, creating an interesting look.

Photo by Blaz Photo on Unsplash

Business Insider via Yahoo Finance

I created a brand new job at Deloitte: corporate influencer. This is why your company needs one too

Lara Sophie Bothur & Polly Thompson

Wired

Forget SEO. Welcome to the World of Generative Engine Optimization

Zoë Schiffer

New York Times

Renting a San Francisco Apartment in the A.I. Boom? Good Luck

Natallie Rocha

404 Media

The AWS Outage Bricked People’s $2,700 Smartbeds

Matthew Gault

Look who's talking 👄

"If you think anybody has the complete answer, or any vendor that’s on this stage says, ‘I know exactly what to do here in the world of this new agentic enterprise,’ they’ve made a mistake."

- Marc Benioff, speaking at the Gartner’s IT Symposium/Xpo this week as reported by CIO Dive.

Please subscribe to the newsletter and feel free to share.

3350 Virginia St, Suite 262, Miami, FL 33133

ForwardThinking 🤔

Let’s play the blame game

Stuff’s gonna happen

Preventing future incidents instead of placing blame

What's new on the blog 📰

Samsara is using AI to steer better driving behavior

Crusoe is building data centers where power is abundant and cheap

How Liberty Mutual’s CIO is leading the shift from AI experimentation to execution

Amazon and Google look to hook enterprise AI buyers with new tools

Becoming an AI-first business requires a total organizational mindshift

As OpenAI becomes the 10,000-pound gorilla, where do startups fit?

News of the Week 📣

Adobe FireFly foundry lets companies tune models for their brand assets

Openai enters new browser wars with release of chatgpt atlas

Anthropic and google announce latest multi-billion dollar Ai-related dalliance

My adventures in AI 🤖

What I'm reading 📚

Business Insider via Yahoo Finance

I created a brand new job at Deloitte: corporate influencer. This is why your company needs one too

Wired

Forget SEO. Welcome to the World of Generative Engine Optimization

New York Times

Renting a San Francisco Apartment in the A.I. Boom? Good Luck

404 Media

The AWS Outage Bricked People’s $2,700 Smartbeds

Look who's talking 👄

Subscribe

FastForward #43: Let’s play the blame game

You may also like...