LLM Phishing A/B Results, Moxel Follow-up, and ThreatWatch Feed Health

LLM phishing A/B test results

Two weeks of data. Here’s what actually happened.

Setup recap (from week 10): Human-crafted phishing templates as control, Claude Sonnet-generated templates as test. Same delivery infrastructure, same target segments, same subject lines. Only variable was the email body.

Results:

Metric	Human-crafted	LLM-generated
Open rate	41%	39%
Click rate	18%	17%
Credential entry	6%	9%

Open and click rates are essentially identical, within margin. The credential entry rate is where it gets interesting: LLM-generated templates outperformed human-crafted by 3 percentage points on the credential entry step.

My read on why: the LLM-generated lures were more contextually specific. The prompt I used included role and department context, which the model used to write copy that felt less generic than our standard templates. The human-crafted templates are good but they’re built for broad applicability. The LLM can tailor cheaply at scale.

This has a few implications worth sitting with. First, the economics of phishing simulation shift: generating 50 customised templates per engagement costs almost nothing in API tokens vs. the time a red teamer would spend writing them. Second, the same capability is available to actual threat actors. The quality bar for social engineering is going to keep rising because the production cost is collapsing.

I’m not going to overstate this. It’s one test, one environment, specific scenarios. But the direction is clear enough to take seriously.

Moxel: where things stand

Following up on last week’s 4-GPU benchmark. A few things I’ve confirmed since:

The prefetch improvement (~11 to ~23 tokens/second at batch size 1) holds across different model sizes. Tested with Llama 3 70B as well as Mixtral and the pattern is consistent. The execution graph predictability for transformers is the key property being exploited.

One thing that surprised me: the bottleneck at low batch size is not always PCIe. Sometimes it’s the CUDA kernel scheduling overhead from coordinating across four devices. Profiling showed about 15% of latency in some configurations is from device sync, not data transfer. Working on reducing the sync frequency.

Next milestone for Moxel is PCIe gen 5 hardware access. That would push theoretical bandwidth from ~28 to ~56 GB/s, which should get single-stream latency much closer to NVLink setups.

ThreatWatch feed freshness now live

The feed health dashboard from week 8 is live. Green/amber/red freshness indicators are showing on all 52 active feeds. Since deployment:

• 3 more feeds have gone amber (not caught before)
• 1 feed we marked red has been replaced with a working alternative
• The .onion source situation is as bad as expected: 4 of our 9 dark web sources have shown at least one >48h staleness event in the past two weeks

The replacement pipeline is still manual. Next step is building a semi-automated feed evaluation process: when a source goes red, automatically surface candidate replacements from a curated list and flag for human review rather than having to identify them from scratch each time.

One thing I’ve been chewing on

The LLM phishing result and the indirect prompt injection research from week 7 are pointing at the same underlying shift. The assumption that humans are the last line of defence against social engineering is being eroded from both directions simultaneously: attacks are getting cheaper and more targeted, and the applications people trust (AI assistants, RAG systems) are themselves vulnerable to injection.

That’s not a new observation at a high level, but the concrete numbers are starting to make it feel less theoretical.

---

Next week: digging into the SoK credential-harvesting paper that’s been in my queue, C2PA 2.1 spec changes for ProvStamp, and a closer look at the ClearFake IOC velocity that’s been showing up in ThreatWatch feeds.