Deep enough to be useful
We study how models interpret meaning and make bounded decisions: ambiguity, reference, context, tool selection and programming workflow choices. The first public prototype starts with conversational meaning, using real saved experiments.
The aim is to connect an aggregate result to its literal examples, calling conditions and failure cases. Developers should be able to inspect the disagreement, not just accept the ranking.
The Somerville Semantic Decision Benchmark
SSDB is the working name for the benchmark being developed here. The current pilot shows separate diagnostic measures. A broader frozen standard and overall score will follow only after its tasks and scoring rules are reviewed.
A living publication
The Somerville Signal will connect research notes, a newsletter and narrated explanations. The planned operation uses Hermes for research and editorial work, VoiceCast Studio for video production, and Starship for long-term organization. Those integrations are planned; this prototype serves measured evidence independently.
Independence and disclosure
Travis’s own model projects, including a future CapCom entrant, would be evaluated under the same frozen rules, with ownership disclosed. This site does not claim to be the first semantic benchmark or to measure intelligence in its entirety.
Read the research method →