Ninja Labz blog

From a laptop on my back to a lab in Suffolk: the day the model came home

The first time ninja.ing served a page, the server was a laptop in a rucksack between Suffolk and Ankara, offline only on the plane. Tonight the same estate serves a 27-billion-parameter model from two cards in a Suffolk lab, wired into our own graph, behind a hosting arrangement nobody would design on purpose. The story, the numbers, and the honest bit about what the model is and is not.

TL;DR. ninja.ing started as a laptop that never got to sleep. Today, Zero Day of a programme we are calling COLD IRON, two RTX PRO 6000 cards arrived in Suffolk and by the evening the Skunkbox was serving multiple models: Qwen3.8-27B to the estate at 78.7 tokens a second, a second copy of it to the public, and Captain Chaos beside them, reading our ransomware graph, our CVE ledger and our own published analysis through a tool layer we built this afternoon. The model is not trained on our data. It reads it. That distinction is the whole design, and the last section explains why.

1. The rucksack years

The first time ninja.ing answered a request from the open internet, the machine answering it was a laptop in a rucksack. It left a village in Suffolk, crossed Europe, and spent three days getting to Ankara. The only time the site went offline in those three days was on the plane, because the airline did not sell Wi-Fi and I did not yet own a satellite dish.

That was not a stunt. It was what we had. The graph lived on that laptop. The ingesters ran on it. The first version of the threat globe rendered on it, badly, in a hotel. If you hit ninja.ing in that window and it was slow, now you know why: the server was in an overhead locker.

What that period taught us is still the spine of how the estate is built. Everything had to survive a lid closing. Every service had to come back on its own. State had to live somewhere that was not RAM. Those three rules are why the estate today runs forty-odd containers across two rented servers, a VPS, a Cloudflare edge and now a lab, and why a cable being pulled in any one of them is a slow request rather than an outage.

2. The hosting arrangement nobody would design on purpose

Here is what a request to Ninja Qwen crosses tonight, in order:

  1. Cloudflare takes the TLS and the abuse.
  2. Box 1, a rented server in Finland, runs Caddy, the Signal graph (Neo4j, 13.7 million nodes), Fusion, and the Cortex “ask” service that holds every read-only graph query the model is allowed to run.
  3. The admin gate on Box 1 asks Ninja Labz, on Box 2 in the same data centre, whether you are signed in and whether you are allowed. Only then does the request reach the chat UI.
  4. The chat UI (Open WebUI, patched) runs on Box 1 and talks to the model over WireGuard, a private tunnel that now has a fourth member.
  5. The models run in a lab in Suffolk on a machine we now call the Skunkbox. Not one model: the 27B reader on card 0 with a 128k window for the desk, a second copy of it on card 1 for the exams and the public door, Captain Chaos (gpt-oss-20b with the satire adapter we trained in October) running on card 1 beside it, and the trained 3B still on a CPU in Finland as the fallback if the lab goes dark. All served by llama.cpp. The box has no public port. It reaches the world through the WireGuard tunnel and a Cloudflare Tunnel, both outbound.
  6. The tools go the other way: when the model decides it needs the graph, the call goes from the UI on Box 1 to the ask service on Box 1, never to the lab. The lab holds weights and does inference. The facts stay in Finland.
  7. Starlink arrived today in a box. When it is on the roof it becomes the second uplink, so a training run does not die with the fibre.

A request therefore crosses the North Sea twice, three jurisdictions, two private tunnels and one public one, and comes back in about a second and a quarter for a graph question. Nobody would draw that on a whiteboard. It is what you get when a thing grows for two years and every piece has to keep working while the next one is bolted on.

3. What arrived, and what it did on day one

A word on the name. Until this morning “the Skunkbox” was the honeypot VPS: the cheap box in a data centre that gets scanned ten thousand times a week and feeds the malware range. The name has moved indoors. The lab machine is the Skunkbox now; the VPS is its sensor in the field, and from tonight everything the sensor catches is mirrored into the Skunkbox’s vault.

Cards 2 x NVIDIA RTX PRO 6000 Blackwell, 96 GB each, owned, on site
Host The Skunkbox: Ryzen 9 9950X3D2, 123 GB RAM, 1.8 TB + 3.6 TB NVMe, Ubuntu 24.04
Models served tonight Qwen3.8-27B twice (desk and public), Captain Chaos (gpt-oss-20b + our adapter) on card 1, the trained 3B on Box 1’s CPU as fallback
Reader of record Qwen3.8-27B (Apache 2.0), the unsloth Q4_K_M GGUF, 16.5 GB on disk, 24 GB on the card with a 128k window, served by llama.cpp
First numbers 78.7 tokens/s generating, 576 tokens/s on the prompt, card at 500 W under load
Card 1 A second copy of the 27B for the exams and the public door, and gpt-oss-20b (GGUF) with our Captain Chaos adapter; the rest reserved for training
Uplinks Fibre now, Starlink being booked

Day one was bring-up, not results. Both cards seen by the driver and by Docker. The data disk formatted and mounted. The box added to the estate’s private network. The data lake mirrored locally. Backups of both servers started flowing to the lab, slowly, because a home broadband line is a home broadband line. One lane-sharing quirk found on the motherboard (the 4 TB drive sits in a slot that halves the second card’s link), which is a ten-minute fix with a screwdriver and a calm moment, neither of which we had tonight.

4. The honest bit: what this model is and is not

It is not trained on our data. Qwen3.8-27B is Alibaba’s stock instruct model, Apache 2.0, with one sentence of ours in its system prompt. We have run seventeen training runs on this estate since October trying to teach smaller models our facts. Every one of them lost to that sentence. The lesson, written into our charter, is that facts live in the graph and the model reads. Anyone telling you their chatbot “knows” their threat intel because they fine-tuned it on it is describing a model that will confidently misremember a victim name in June that it learned in May.

What it does have is tools. Seventeen methods in the chat UI and, as of this afternoon, a second door: an OpenAPI tool server with twenty read-only routes over the graph. Victims by crew, sector, country and date. CVEs by product, weakness, KEV date and score. Actor profiles, shared tooling, the shortest path between two names. Full-text search across every dossier. Full-text search over our own published analysis, so the model can say what we said, dated. Every one of those is a fixed query template. The model cannot write a line of Cypher. It cannot write to anything. The ask service that answers it has no route that is not a read.

And it has a web search, which the graph tools do not replace: the graph is what we have observed and ingested; the web is what the world is saying right now. The system prompt tells the model which to trust for what.

5. The gate, and what it caught

We have a rule on this estate that nothing goes public until it has passed a test that could have failed it. Tonight the test was the same 200 hand-written questions we score every reader on, run through the live path from Finland to the new card. The bar was 197 out of 200, which is what this model scored on a rented card on 2 November, plus or minus two.

It scored 192. Twice, on two different windows, the identical eight misses.

The post-mortem took twenty minutes and the fault was ours: the harness script had never forwarded the one sentence that lifts this model from 193 to 197, so both runs scored the model without it. One line fixed it. The third run scored 196 of 200, inside the bar, and the diff against the 2 November run is three questions lost, two gained, one still missed by both. There is a footnote worth keeping: the model file upstream was re-quantised on 11 November, so the exact bytes that scored 197 no longer exist to download. 196 on this file, with its hash written next to it, is the baseline our own hardware starts from.

That is what a gate is for. A number you cannot reproduce is not a number, and a number you can is worth the twenty minutes.

6. Where to hear it

Ninja Qwen is at qwen.ninja.ing, behind a Ninja login, with the graph tools attached. Try it yourself, no login: gpu.skunkworkz.io is the same 27B on the Skunkbox’s second card, the plain llama.cpp chat page, rate-limited per address. It has no graph tools and no web search; that is the model on its own, which is the honest baseline for everything above. It stays up while the electricity bill stays sane. The build log, day by day, is on skunkworkz.io. The graph it reads is Ninja Signal; the sandbox next to it is here.

The models, for the record: Qwen3.8-27B · Qwen3.8-27B GGUF (unsloth) · gpt-oss-20b · gpt-oss-20b GGUF (ggml-org) · llama.cpp · Open WebUI.

From a rucksack to the Skunkbox is not a victory lap. It is the same site, with the server finally allowed to sit still.

ScottG · Ninja Labz · ninjalabz.io

ninja qwenninja cortexlocal llmrtx pro 6000llama.cppwireguardcloudflare tunnelstarlinkself-hostingthreat intelligenceopen webuiqwen3.8

Every morning at 06:00 UK an AI reads every threat-intel item from the last 24 hours and writes the paper, with receipts.

Read Probably Fine Daily