UI generation eval report

JudgmentKit UI-Generation Eval

Live provider-generated paired UI evidence for dated standalone artifacts. Use this report to review winner, delta, leakage, activity-fit evidence, and generation provenance by case.

Latest run outcome

2/2 cases passed

2 guided wins, 0 baseline wins, 0 ties.

Claim level
repeated_pair_signal
Run date
2026-07-03
MCP release
0.6.5
Run
run-002
Eval id
judgmentkit-ui-generation-paired-artifact-v1
Metric scale
0-5
Generation
live_provider
Publication
publishable

Generation policy

Live provider artifacts

Calls a live provider for each baseline and JudgmentKit-guided variant, writes dated generated HTML artifacts, then scores those artifacts.

Mode
live_provider
Provider
codex
Model
gpt-5.5
Publishable
publishable
Case Variant Artifact Generated
refund-triage-standalone-v1 Raw baseline 2026-07-03/mcp-0.6.5/run-002/generated-artifacts/refund-triage-standalone-v1/baseline.html 2026-07-03T17:24:34.697Z
refund-triage-standalone-v1 JudgmentKit guided 2026-07-03/mcp-0.6.5/run-002/generated-artifacts/refund-triage-standalone-v1/guided.html 2026-07-03T17:26:07.483Z
dinner-playlist-standalone-v1 Raw baseline 2026-07-03/mcp-0.6.5/run-002/generated-artifacts/dinner-playlist-standalone-v1/baseline.html 2026-07-03T17:28:27.031Z
dinner-playlist-standalone-v1 JudgmentKit guided 2026-07-03/mcp-0.6.5/run-002/generated-artifacts/dinner-playlist-standalone-v1/guided.html 2026-07-03T17:35:40.489Z

Calls a live provider for each baseline and JudgmentKit-guided variant, writes dated generated HTML artifacts, then scores those artifacts.

Qualitative paired-artifact evidence from live provider-generated artifacts only; not a statistically powered benchmark.

Case review

Refund triage handoff

Review the selected refund request and prepare the next handoff.

Passed
Winner
JudgmentKit guided
Expected winner
JudgmentKit guided
Score delta
+36.08
Threshold
20
+36.08 guided delta

Visual evidence

Initial viewport screenshots captured from dated live-generated artifacts. Visual evidence is not used for scoring.

Mobile screenshots

Metric comparison

Baseline and guided scores use the 0-5 metric scale; totals remain 0-100 weighted.

Metric Baseline Guided Delta
Activity Fit
1/5 1 present, 4 missing
5/5 5 present, 0 missing
+4
Decision Support
3.75/5 3 present, 1 missing
5/5 4 present, 0 missing
+1.25
Disclosure Discipline
4.25/5 1 leaks
5/5 0 leaks
+0.75
Handoff Completeness
3/5 3 present, 2 missing
4/5 4 present, 1 missing
+1
Task Success Support
1/5 1 present, 4 missing
3/5 3 present, 2 missing
+2
Confidence Rework Signals
1.67/5 1 present, 2 missing
5/5 3 present, 0 missing
+3.33

Activity-fit evidence

1 to 5 matched terms

Guided output surfaced more of the task vocabulary reviewers need to judge activity fit.

Baseline matched (1)
  • Evidence checklist
Guided matched (5)
  • Daily triage
  • Refund escalation queue
  • Customer refund escalation
  • Evidence checklist
  • Policy review context
Guided missing (0) None

Implementation leakage

1 leaks to 0 leaks

Leakage findings count terms that make implementation mechanics visible in the primary artifact.

Baseline leakage (1 leaks)
  • JSON schema
Guided leakage (0 leaks) None
Expected outcomes and rationale

Expected outcomes

  • The reviewer can identify the selected case.
  • The reviewer can choose approve, policy review, or return for evidence.
  • The reviewer can complete a handoff with a reason and next owner.

Rationale

  • JudgmentKit-guided artifact scored 36.08 points above baseline.
  • Implementation leakage changed from 1 baseline terms to 0 guided terms.
  • Activity-fit evidence changed from 1 matched terms to 5 matched terms.

Case review

Dinner playlist builder

Build a 10-song dinner playlist that starts mellow, lifts in the middle, avoids disliked artists and explicit tracks, and leaves a sequence note.

Passed
Winner
JudgmentKit guided
Expected winner
JudgmentKit guided
Score delta
+41.65
Threshold
20
+41.65 guided delta

Visual evidence

Initial viewport screenshots captured from dated live-generated artifacts. Visual evidence is not used for scoring.

Mobile screenshots

Metric comparison

Baseline and guided scores use the 0-5 metric scale; totals remain 0-100 weighted.

Metric Baseline Guided Delta
Activity Fit
1.67/5 2 present, 4 missing
5/5 6 present, 0 missing
+3.33
Decision Support
2/5 2 present, 3 missing
2/5 2 present, 3 missing
0
Disclosure Discipline
0/5 11 leaks
5/5 0 leaks
+5
Handoff Completeness
3.75/5 3 present, 1 missing
3.75/5 3 present, 1 missing
0
Task Success Support
3.57/5 5 present, 2 missing
3.57/5 5 present, 2 missing
0
Confidence Rework Signals
0/5 0 present, 3 missing
3.33/5 2 present, 1 missing
+3.33

Activity-fit evidence

2 to 6 matched terms

Guided output surfaced more of the task vocabulary reviewers need to judge activity fit.

Baseline matched (2)
  • Guest preferences
  • Sequence note
Guided matched (6)
  • Dinner brief
  • Guest preferences
  • Suggested tracks
  • Playlist sequence
  • Conflict checks
  • Sequence note
Guided missing (0) None

Implementation leakage

11 leaks to 0 leaks

Leakage findings count terms that make implementation mechanics visible in the primary artifact.

Baseline leakage (11 leaks)
  • data model
  • track table field
  • JSON schema
  • prompt template
  • tool call
  • resource id
  • API endpoint
  • CRUD
  • field
  • model
  • candidate
Guided leakage (0 leaks) None
Expected outcomes and rationale

Expected outcomes

  • The host can assemble a 10-song playlist.
  • The host can catch explicit and disliked-artist conflicts.
  • The host can explain the sequence from mellow start through lifted middle to soft close.

Rationale

  • JudgmentKit-guided artifact scored 41.65 points above baseline.
  • Implementation leakage changed from 10 baseline terms to 0 guided terms.
  • Activity-fit evidence changed from 2 matched terms to 6 matched terms.