liminal

The measurement problem

LLM-as-judge is how most teams evaluate chatbots, summarizers and RAG pipelines. The trouble is that a judge is a biased instrument: it rewards verbosity, drifts from human taste, and disagrees with other judges. Most eval tooling reads its scores as ground truth anyway.

liminal fits a statistical model of the judges instead, so the output separates who is harsh, who is noisy, and what quality actually looks like underneath.

What the model says

liminal fits a Graded Response Model, which treats a 1–5 rating as an ordered category rather than a number to average. The probability that judge j rates item i at least k is:

P(rating_ij ≥ k) = σ( a_j · ( θ_i − β_j + γ_j·z_i − τ_jk ) )

θ_i

Latent true quality of item i, the thing you actually wanted to measure.

β_j

Severity of judge j. Higher means a harder grader.

a_j

Discrimination of judge j. Higher means more reliable.

γ_j

Length-bias coefficient. Positive means the judge rewards verbosity.

z_i

Standardized log-length of item i, the covariate that carries the bias.

τ_jk

Judge j’s k-th category threshold. Four of them for a five-point scale.

Separating γ from θ is what pulls “long” apart from “good”, which is the best-documented failure mode these judges have.

How it is estimated

Item quality is integrated out by marginal maximum likelihood over a 20-node Gauss–Hermite quadrature, and the judge parameters are optimized with L-BFGS-B. Gradients come from JAX autodiff. The IRT core is written from scratch rather than pulled from an existing package.

Four identifiability constraints make the parameters mean something rather than merely fit: severities sum to zero, log-discriminations sum to zero, thresholds are centred per judge so severity is not double-counted, and thresholds are built from positive increments so they stay ordered.

Item quality then comes back as a posterior mean by the same quadrature, with credible intervals approximated by sampling.

Reproducible by construction

Every LLM response is content-addressed on a SHA-256 of the provider, model, prompt, temperature and token limit, then written to a sharded cache. A study reproduces exactly even after the underlying models have moved on, and re-running one costs nothing. Every source of randomness is seeded explicitly.

One OpenAI-compatible client covers Groq, Cerebras, Gemini and OpenRouter, with bounded concurrency and exponential backoff with jitter on retryable status codes.

What has been checked

59

Unit tests across core types, matrix, metrics, providers, judges and datasets.

ρ > 0.70

Spearman correlation for recovered item quality ordering, at 5 judges and 400 items, against simulated ground truth.

ρ > 0.9

Spearman correlation for recovered judge severity, at 4 judges and 600 items, against simulated ground truth.

1e−9

Agreement between this Krippendorff α implementation, written from first principles, and the reference package.

Recovery accuracy tracks the information ceiling rather than beating it, and adding judges strictly improves it. Those figures come from simulation, where the true parameters are known. SummEval is the benchmark the flagship run will use: 100 CNN/DailyMail articles across 16 summarization systems, three expert annotators each.

Where it stands

Read the code

Apache-2.0, built on free-tier models and a public benchmark so anyone can reproduce the result rather than take it on trust.

View source