Evaluation harness budget

평가 하네스 예산

Fix tool, attempt, and time budgets so an agent evaluation score is interpretable.

···
html
<div class="stage"><div class="title">EVAL BUDGET</div><div class="canvas"><div class="budget"><span>attempts</span><div class="ticks" id="budgetTicks"><i></i><i></i><i></i><i></i></div><b id="budgetNum">1</b></div><div class="budgetChart"><div><span>SOLVED</span><i id="solved"></i><b id="solvedN">2/10</b></div><div><span>COST</span><i id="cost"></i><b id="costN">1×</b></div></div><div class="budgetNote">same tasks · same scoring</div></div></div>
css
.stage{width:min(94vw,820px);height:min(88vh,330px);box-sizing:border-box;padding:clamp(9px,2.4vmin,18px);border:1px solid var(--line);border-radius:13px;background:var(--surface);display:flex;flex-direction:column;gap:clamp(7px,2vmin,14px);font:600 clamp(12px,2.5vmin,16px)/1.28 "Pretendard Variable",Pretendard,-apple-system,BlinkMacSystemFont,"Apple SD Gothic Neo",sans-serif;color:var(--fg);overflow:hidden}.title{color:var(--accent);letter-spacing:.035em;flex:none}.canvas{position:relative;flex:1;min-height:0;display:flex;align-items:center;justify-content:center}.muted{color:var(--muted)}.mono{font-family:ui-monospace,SFMono-Regular,monospace}.box{border:1px solid var(--line);border-radius:8px;background:var(--bg);padding:clamp(5px,1.4vmin,12px);text-align:center}.active{border-color:var(--accent)!important;color:var(--accent)!important;background:color-mix(in srgb,var(--accent) 12%,var(--surface))!important}.canvas{flex-direction:column;gap:clamp(7px,2vmin,13px)}.budget{display:flex;align-items:center;gap:7px;width:min(94%,530px)}.budget span{width:clamp(56px,16vmin,80px)}.ticks{display:flex;gap:4px;flex:1}.ticks i{flex:1;height:clamp(13px,3vmin,23px);border:1px solid var(--line);border-radius:3px}.ticks i.on{background:var(--accent);border-color:var(--accent)}.budgetChart{width:min(94%,530px);display:flex;flex-direction:column;gap:6px}.budgetChart>div{display:flex;align-items:center;gap:6px}.budgetChart span{width:clamp(56px,16vmin,80px)}.budgetChart i{height:clamp(11px,2.7vmin,18px);border-radius:3px;background:var(--accent);transition:width .4s}.budgetChart b{margin-left:auto}.budgetNote{color:var(--muted)}
js
let n=0;function budget(){const step=n++%4;document.querySelectorAll('#budgetTicks i').forEach((el,i)=>el.classList.toggle('on',i<=step));document.getElementById('budgetNum').textContent=step+1;document.getElementById('solved').style.width=[20,40,58,70][step]+'%';document.getElementById('cost').style.width=[18,35,54,72][step]+'%';document.getElementById('solvedN').textContent=[2,4,6,7][step]+'/10';document.getElementById('costN').textContent=(step+1)+'×'}budget();setInterval(budget,650)

Agent evaluation results depend on the harness and the effort budget allowed. For a controlled comparison, fix the tasks, scoring, and budget, then report the tools and cost.

The demo changes the attempt budget and shows both solved cases and cost rising. Interpret a score as performance under its stated conditions, rather than an unconditional model ranking.

When to use

Use it when comparing agent models or prompts or publishing evaluation scores.

Open as page ↗