1. I’m not training a model from scratch

When I say I’m building my own AI system, I don’t mean training a large language model from scratch. That would need massive datasets, special hardware, and a whole different level of investment.

Instead, I’m building the environment around an existing, powerful model: what it can access, what rules it follows, how it stores and retrieves information, and which parts of my work can use it.

Most of the real engineering happens in this surrounding system, and that’s where I’m focusing my efforts.

2. Why build AI locally

Cloud AI is very convenient, and I still use it every day. But it means my material has to leave my computer before anything useful happens, and I’m subject to someone else’s pricing, limits, and policy changes.

Running a model on my own hardware removes that dependency. Cloud AI isn’t bad, but having the choice to avoid it matters.

The ongoing costs include electricity, maintenance and the initial hardware investment, rather than a usage-based inference bill.

3. The hardware becomes the AI server

I’ve bought a Mac Studio with an M5 Ultra chip and 96GB of unified memory for this project.

I’m setting it up as a dedicated AI machine, not for everyday tasks. My regular laptop will still handle email, browsing and admin.

The Studio’s only job is to run the model and the retrieval system, accessible over my local network.

Keeping them separate means long AI tasks won’t slow down the computer I use daily.

4. Why unified memory matters

Large language models are partly measured by parameter count. For example, a 70-billion-parameter model has about that many adjustable values that shape how it understands language, and more parameters usually require more memory.

Most computers split memory between the CPU and a much smaller GPU pool. Apple Silicon’s unified memory lets the CPU, GPU and Neural Engine share one large pool.

That makes it possible to run much larger models locally than would be practical with a small dedicated GPU memory pool.

For this work, memory isn’t just a specification. It helps determine which models I can realistically run.

5. The model is only one part of the system

Just downloading a capable model doesn’t give you a usable private AI platform.

The model handles reasoning, but everything else, including retrieval, embeddings, storage, rules, access control and automation, still needs to be built around it.

I see it as layers: the language model, a retrieval layer to find relevant material, an embedding layer to make that material searchable by meaning, a storage layer to hold it, and an instruction layer to set the rules for how everything works.

6. Running the model locally

I plan to run the model with Ollama, a local runtime that loads it into memory and lets applications communicate with it consistently.

Ollama isn’t the model itself. It’s the layer between a request and the model’s reasoning.

This means a local script or application can send it a task and receive a result without relying on a hosted model API for the inference.

7. The model still doesn’t know my own material

A strong general-purpose model knows a lot about many subjects, but it knows nothing about my private documents simply because I have downloaded the model.

It also cannot automatically know information created after the data used to train it.

Retraining it every time something changes isn’t practical.

The solution I’m exploring is retrieval-augmented generation, or RAG. Instead of changing what the model has learned, the system retrieves relevant material and provides it when the model needs to answer a question.

8. RAG gives it grounded, current knowledge

The idea is simpler than it sounds.

Imagine a skilled consultant who hasn’t seen your files. Before you ask them anything important, someone brings the relevant documents and puts them on the desk.

RAG does something similar automatically. When a question comes in, the system searches a knowledge base for relevant material and gives that evidence to the model alongside the question, instead of relying entirely on what the model already knows.

Think of a small organisation with years of internal procedures, templates and past reports.

If you ask, “What do we normally do before starting a new project?”, the aim is for the system to retrieve that organisation’s own process first instead of relying on a generic answer.

9. Embeddings, vector databases and chunking

To find the right material by meaning rather than only by exact words, the system needs a way to represent and compare semantic similarity.

An embedding model turns a piece of text into a numerical representation. Text with similar meanings can then be identified as being semantically related even when it uses different words.

A vector database can store these representations and retrieve close matches when a new question arrives.

Large documents can also be divided into smaller chunks. Each chunk receives its own embedding, allowing the retrieval system to return the passages most relevant to the question rather than sending an entire document to the language model.

10. Rules and structured output

Giving a model access to real material doesn’t make it reliable on its own.

It needs clear rules: use the evidence provided, flag missing information instead of making it up, separate facts from recommendations, and cite sources where appropriate.

For other software to use the output, free-form paragraphs are not always enough.

The response can instead use structured data with defined fields so another program can store, display or process the result more reliably.

11. The model, and why

I’m planning to use DeepSeek-R1-Distill-Llama-70B, an open-weight reasoning model distilled from a much larger original model.

I chose it for a few clear reasons.

The weights are downloadable rather than being available only through a hosted API. It is designed for reasoning-heavy tasks, and a 70-billion-parameter model is large enough to make this experiment meaningful while still being realistic to explore on a high-memory Mac Studio when appropriately quantised.

It isn’t the smallest or fastest option, and the real test will come when I can measure its performance on the new hardware.

12. Understanding quantisation

A 70-billion-parameter model stored at 16-bit precision needs roughly 140GB for its weights alone before allowing for runtime overhead.

That is beyond the 96GB memory configuration I have chosen.

Quantisation helps by representing the model’s parameters using fewer bits.

The parameter count and the precision are two different things. The 70B figure describes approximately how many parameters the model contains. Terms such as 4-bit and 8-bit describe how those values are represented.

A 4-bit quantised 70B model therefore remains a 70B-parameter model. The representation of its weights has been compressed so that it requires substantially less memory.

The quantisation format I plan to test is Q4_K_M. Rather than assuming the theoretical numbers will translate directly into good real-world performance, I want to test memory use, speed, context length and answer quality once the hardware arrives.

13. The software stack

Two main software tools are planned for the first version of the system.

Ollama will provide the local model runtime. AnythingLLM is the interface I intend to use for day-to-day interaction, document ingestion and retrieval.

The attraction is that I don’t need to build every component from zero.

Part of my approach is to understand and combine good existing tools on hardware I control, then build additional components only where they add genuine value.

14. Automation, with a human still in the loop

Once retrieval and generation work reliably, new material could be processed and added to the knowledge base without retraining the language model.

That creates opportunities for automation, but I don’t want to build a system that quietly makes significant decisions on its own.

My design principle is to keep a person involved where judgement matters.

The system can retrieve evidence and suggest an answer, but the supporting evidence should remain visible and the output can be reviewed before another action follows.

15. Access control has to happen before retrieval

If the system eventually serves more than one person or group, their information must remain properly separated.

One person’s private documents should never appear in somebody else’s retrieval results simply because the underlying search considered them semantically relevant.

That means the retrieval design needs to take authorisation into account before material is returned to the model.

Security cannot simply be treated as a filter applied after an answer has already been generated.

16. Local doesn’t mean rejecting cloud AI

I’m not saying everything should run locally.

Cloud models can be faster, easier to scale and more capable for particular tasks. I still use them when they are the better fit.

What I want is choice.

Appropriate private work can stay on hardware I control, while other tasks can use a hosted model when the benefits justify it.

The point isn’t to turn local AI into an ideology. It is to stop treating the cloud as the automatic answer to every AI problem.

17. Where I am right now

The Mac Studio is expected at the end of November, so right now is the preparation phase rather than the testing phase.

I’m using these weeks to get the fundamentals in place before the hardware arrives: understanding the tools I intend to use and working out how I want to organise the material I will eventually bring into the system.

I have a sizeable body of my own notes and reference material to bring in, and getting that structure right now should save a much bigger mess later.

I’m also writing things down as I go, partly so I don’t lose my own reasoning a few weeks from now, and partly because I’d rather be honest about the process, including what I get wrong, than only appear once everything works.

18. What I’m really trying to own

This isn’t really about just running a model on a powerful computer.

It’s about understanding enough of the stack, including the model, the material that grounds it, where it runs, and the rules it follows, to make deliberate choices about each part instead of simply accepting whatever a platform gives me.

I’m not ready to say exactly what I’ll build on top of this yet. It is still taking shape, and I’d rather show real progress than talk about an idea before it exists.

The machine arriving next month is the first real step.

I’ll keep documenting the process as I learn what works, what doesn’t, and what it really takes to build education-focused AI on infrastructure I control.