Why I Run Some AI Locally and Some in the Cloud

The machine on the desk and the cloud behind it, doing different jobs.

As someone who has spent almost every day playing with AI in one way or another since GPT 3.5, I made a decision a long time ago to have a mix of local and cloud based LLMs. There's a few reasons for it but it all boils down to how much you value your time.

Let's get into it.

AI Does Not Give You More Time

Have you ever heard the phrase "Linux is free as long as you don't value your time"? Unfortunately I didn't hear that phrase until just a while ago.

But here's what that phrase means:

If you don't care about your time, an operating system that's completely free actually costs you nothing.

But if you DO value your time, a free operating system cost you EVERYTHING.

The Value of Time in an AI Automation Calculation

I had a boss years ago tell me "You have to figure out how much you are willing to spend in a month for something to no longer be a problem you have to deal with".

At the time, I chalked this up to a boss who had too much money giving bad advice to a college student who had nothing but time, and no money to offload their projects.

But now, especially as I work so much with AI, that calculation is at the forefront of my mind every day.

Because AI is freaking amazing. The amount of work I can automate and offload to a separate intelligence system is astounding. But it's also really, really easy to get caught up in a loop of automations that go nowhere.

So I decided I would make an intentional decision to split my time with AI and Non AI projects.

Splitting Your Time With AI and Non-AI Projects

So what is your time worth? To me, I estimate an hour of my time is worth roughly $65 to $150 depending on what client work I have on my plate that day.

So to me, there's actually two questions that I need to answer before I start working on a thing with AI.

1. Will the AI be FASTER than I am at this particular task in the next hour.
2. Will the QUALITY of its work be better than my own in that hour.

If you're like me, then half the time you ask an AI to do a task for you, it actually takes longer to setup the AI to succeed than it would take to accomplish the task.

So where does AI fit into my workflows?

My AI Framework for Task Automation

Quite simply, I have turned myself into a QA fixer for everything that goes out for my clients. I usually build the first asset for any client by hand the first time. Then, once I believe the deliverable is perfect for that niche, I then turn to an automation system building to make it easy to automate that system multiple times over.

This system is by no means perfect. But it does enable me to be much leaner and more effective with my work. It also gives me a level of control over the deliverables.

It worked for my day job, it's worked for clients that work with billion dollar corporations, and it's worked in my personal automation pipelines.

So now to the crux of the post. How does Local AI fit into my workflows, and it ultimately comes down to the mistake most business owners have made at least once in their lives.

Why Anything Runs Locally at All

Overreliance.

Maybe the most overlooked issue businesses have is solving the 'Hit By a Bus' problem.

In business staffing, it's making sure employees don't just build amazing things, but that they build amazing things that can be taught and shared with other employees.

Because just like there's such a thing as technical debt in a codebase, there's also such a thing as over reliance on tools. This is quite literally called the 'Bus Factor', but it is usually considered more applicable for personal turnover rather than technical turnover.

What Happens When Your AI Gets 'Hit By a Bus'?

Now we just extend the same logic from personnel to AI intelligence. What happens in your business TODAY if for some reason, all AI goes away.

For most of us, all that really changes is that the pace of our business slows down.

But for some of us, that's catastrophic.

For example, do you know how to make edits to your website if AI goes down?

Do you know how to write a good blog post?

Do you know how to make an ad?

Do you know how to design a PRD?

Do you know the best methodology for running deep research on a given problem using business data and publicly available data?

Look, you get my point. If you answered 'NO' to any of the above questions, then you are in big trouble if AI gets 'Hit By a Bus'.

That metaphor might be a bit too convoluted so let's bring it down to earth a bit.

What happens if and when the cost of AI compute goes up so high you can no longer justify the price of AI in your business. What if Anthropic or ChatGPT decide tomorrow that your business is one that they don't want to work with to avoid regulatory issues?

There's real world risk to losing your AI if you become overreliant on it. And THAT'S the reason to invest in local models.

Because at the end of the day, they are dumber, slower, and more immediately expensive than frontier models you can rent access to. But they do belong to your business.

Let's talk more about how dumb local AI models can be.

What Local Models Are Not

In my experience, LLMs you can run on your own consumer hardware are very bad compared to other models. However, they can be fixed! So you should know the problems before you try and build with them.

Here's the main problems I see today with local LLMs:

1. They're dumb. They are usually a year behind frontier models intelligence, so whatever Claude or ChatGPT is capable of today, you might be able to replicate it with local LLMs a year from now.
2. They're slow. Local LLMs run on slower hardware. And I don't mean that your Mac Studio with 512gb of ram is slow. I just mean that you don't have access to the crazy fast GPUs in bulk that enterprise datacenters have access to. If you've never run a local LLM before, they are MUCH slower than enterprise grade LLMs.
3. They're clunky. Local LLMs are only HALF the battle. They also are usually missing harness features that come native with subscriptions to Claude Code, or OpenAI's Codex.

All of this boils down to the time thing I mentioned before hand.

If you value your time, Local LLMs are NOT worth the investment. But if you value your ability to stay independent and let your business depend on LLMs you design, then they ARE worth the investment.

So What Are Local LLMs Really?

Local LLMs are an insurance policy. The point of a local LLM is that you have the control over the harness, the model in use, and how it's weighted.

That way, the next time Claude or ChatGPT goes down, you're not left sitting on your hands wondering what you should do now.

And let me be really clear about this. Local LLMs take a LOT of tinkering to be worth it. Deciding on harnesses, building custom tools, developing skills, plugins, and connections, it takes a LOT of time.

But as I learned building my first AI system (B0timus Prime), it can be immensely worth it.

Today, B0timus helps me run lots of cool things. LinkedIn posts from my personal experience, research on things I don't want to give away sensitive data to, client work that needs to be kept private on local servers.

It's a MASSIVE undertaking, but now that I'm more than a year into it, it's starting to pay off.

How to Decide for Your Own Business

So if you read all of that and are STILL interested in running your own LLMs locally for your business, here are the three questions you need to ask yourself to get things started:

1. Where does your most essential data live?
2. Where does your consistent repetition work start and end?
3. What are the core decisions that are made during that process.

The most important question to ask yourself is where does your business (or client) data live. Unfortunately, AI is only as good as the data you feed it. And that works in both directions. If you feed it TOO much data, it loses the plot and will fail consistently.

But if you feed it not enough data, it will starve on essential information and fail to provide good content to you.

So the key to managing a great LLM is actually to find the balance between giving it enough data to do the job well, while also making sure to remove unnecessary datapoints that only serve to overwhelm your context windows.

How to Balance AI, Quality Assurance, and Great Work

So this is more of an evolving question and will always depend more on what work you do, versus what makes sense for everyone.

But in my experience, especially with frontier models, is it's much easier to starve them of essential data than it is to overwhelm them.

So for most people, the best place to start is by connecting any data points you feel are essential for your local (or frontier) model to know what your business is, what it does, and where its gaps are.

For me (and most business owners I know), the place where the best and most recent context is actually your email.

Now I know that I said that and half of you reading are like

"Oh no, goodness me! I can't give it my email!"

And to you, I say "Why not?!?"

Look, your email is not sensitive data the same way invoicing and credit card data are. It's also not sensitive the same way that payment data is private.

But at the same time, it does contain essential data points. Things like conversations about invoicing, conversations about common problems your business experiences, and if you use a transcription service, it also usually contains those transcriptions too.

So an email and whatever system you use to communicate with your team (Slack, Microsoft Teams etc.) is a perfect hub for you to start giving your local or frontier model essential context it needs to truly be helpful.

Should You Hire Someone to Do Your AI Automations for You?

Look, if all of this felt really complex, it might be worth bringing on an AI automation consultant to help parse through and balance the data you feed LLMs so your LLMs truly work for you.

AI automation engineers are usually the most up to date about how AI works underneath the hood, and can help you build your own little lego infrastructure of different LLM tools, context, data, and systems you actually need to make an AI work for you.

If you're interested, I take on a handful of AI automation clients every year and you can take the first step by booking a free consultation with me here.

Thanks for reading.

Signing off,

Preston