Computer use agent

컴퓨터 사용 에이전트

An agent that reads a screen and uses clicks and typing to work through a UI.

···
html
<div class="scene"><div class="heading">OBSERVE → ACT → VERIFY</div><div class="mock-window"><div class="mock-bar">가상 설정 화면 <span id="view-state">관찰</span></div><div class="mock-row">알림 <span id="toggle" class="toggle"><i></i></span></div><div id="cursor" class="cursor"></div></div><div class="muted" id="computer-note">화면을 읽고 다음 동작 결정</div></div>
css
.scene{width:min(94vw,800px);height:min(88vh,326px);padding:clamp(10px,2.7vmin,19px);border:1px solid var(--line);border-radius:13px;background:var(--surface);font:500 clamp(13px,2.4vmin,17px)/1.3 var(--font-sans,sans-serif);position:relative;overflow:hidden}.scene .mono{font-family:ui-monospace,SFMono-Regular,monospace}.scene .muted{color:var(--muted)}.scene .accent{color:var(--accent)}.scene .heading{font-weight:750;color:var(--accent);margin-bottom:clamp(5px,1.6vmin,12px)}.mock-window{position:relative;height:68%;border:1px solid var(--line);border-radius:7px;background:var(--bg);overflow:hidden;margin-bottom:6px}.mock-bar{padding:clamp(6px,1.7vmin,12px);border-bottom:1px solid var(--line);display:flex;justify-content:space-between}.mock-bar span{color:var(--accent)}.mock-row{display:flex;align-items:center;justify-content:space-between;margin:clamp(8px,2.5vmin,18px) clamp(9px,3vmin,20px)}.toggle{width:clamp(35px,11vmin,54px);height:clamp(20px,6vmin,28px);border-radius:99px;border:1px solid var(--line);background:var(--surface);padding:2px;transition:background .3s}.toggle i{display:block;width:45%;height:100%;border-radius:50%;background:var(--muted);transition:transform .3s}.toggle.on{background:var(--accent)}.toggle.on i{background:var(--bg);transform:translateX(100%)}.cursor{position:absolute;width:0;height:0;border-left:12px solid var(--accent-2);border-top:17px solid transparent;right:34%;top:55%;transition:right .4s,top .4s;filter:drop-shadow(1px 1px 0 var(--fg))}.cursor.at-toggle{right:7%;top:25%}
js
const states=[['관찰','화면을 읽고 다음 동작 결정'],['클릭','알림 토글을 누름'],['확인','새 화면에서 켜짐 확인']];let step=0;function draw(){document.getElementById('view-state').textContent=states[step][0];document.getElementById('computer-note').textContent=states[step][1];document.getElementById('cursor').classList.toggle('at-toggle',step>0);document.getElementById('toggle').classList.toggle('on',step===2);step=(step+1)%3}draw();setInterval(draw,700)

A computer use agent receives a screenshot and proposes actions such as clicks and typing for an application to execute. The application returns a new screenshot so the agent can inspect the result.

The demo observes a fictional settings screen, clicks a toggle, then checks the changed state. Coordinate mistakes and unexpected UI states call for an isolated environment and separate approval for sensitive actions.

When to use

Use it for UI work without a suitable API when the result must be verified on screen.

Open as page ↗