From past chats

Turn model incidents into executable fixtures

The new misalignment disclosures are more useful as test material than as six stories to read once. Extract each case into a tiny fixture with the model’s assigned goal, the opportunity it found, the action it attempted, the signal that exposed it, and the containment that should have stopped it. Then replay the closest two against one of your own agent harnesses in a no-network sandbox. The useful follow-up question is not “would my model misalign?” but “which inexpensive trace or policy check would make this class of behaviour visible before an external side effect?”

• OpenAI’s model-misalignment reporting framework
• MITRE ATLAS
• NIST AI Risk Management Framework

Write evaluations as challenge statements

The Millennium Problems for Biology catalogue is an interesting model for software and agent evaluation because each problem names an observable success condition, disallowed shortcuts, controls, and often a blinded or preregistered test. Try writing one “challenge statement” for a recurring agent task: perhaps 100 unseen posts, a fixed label set, no prompt edits after seeing the holdout set, and explicit thresholds for precision, abstention, cost, and latency. That would make a much stronger acceptance test than a handful of favourable examples, even if the resulting benchmark stays private.

• The Millennium Problems for Biology
• OpenAI’s evaluation guide
• OpenAI Evals

From the front page

Trial a classifier in shadow mode

A post-category classifier is a neat place to test whether a decision-focused model genuinely improves on a conventional LLM with structured output. Start with one narrow decision such as Outside, label a small historical sample yourself, and run both approaches without changing any posts. Record the predicted label, confidence, latency, cost, and disagreements; only consider automation after choosing an abstention threshold from the holdout results. Jev’s speed and calibration claims are interesting, but treating them as hypotheses will produce a more useful answer for your actual archive.

• TypeSafe AI’s introduction to System One models and Jev
• OpenAI Structured Outputs
• Scikit-learn’s probability-calibration guide

Learn the grid through one week of Victorian data

A compact way into transmission and distribution engineering would be to follow Victoria’s grid for one week rather than beginning with a textbook. Capture five-minute demand, fuel mix, price, and interconnector flow; annotate the solar ramp, evening peak, low-renewables periods, and any constrained flows. Then read the matching sections of the 2026 Integrated System Plan and Victorian planning report to distinguish an energy shortage from a network, stability, or timing problem. The result could be a small Go data notebook and a short explainer answering: what did the grid need that generation alone could not provide?

• AEMO’s NEM data dashboard
• AEMO’s 2026 Integrated System Plan
• VicGrid’s 2026 Victorian Annual Planning Report