presentsUNAIVERSE
The UN AI timeline

    Learn AI /AI Agents and Automation /Computer use

    Computer use

    Also called: browser use, screen control.

    Detail

    In five seconds

    What it is

    A system capability that lets software observe what appears on a screen and operate apps by clicking, typing, scrolling and selecting items.

    In plain words

    It lets an AI-enabled system operate software the way a person would: by looking at the screen and taking small actions such as clicking, typing and scrolling until the task is done or a human needs to step in.

    An everyday example

    A warehouse supervisor hands a clerk a paper manifest and asks, "Check whether these 24 cartons arrived." The clerk looks at the labels on each box, marks matches on the list, walks to the next pallet, and asks the supervisor when a label is torn or a carton count does not match. If the loading area changes or boxes are stacked differently, the clerk must look again before making the next mark. At the end, the result is not a speech about the shipment but an updated checklist and a note of any exceptions.

    Where the story stops working

    A human clerk brings broad background knowledge, stable judgement and physical awareness. Computer use systems do not inspect a real warehouse and may rely only on pixels or interface metadata from a screen session. Also, software actions can happen much faster and can repeat mistakes at scale, which makes supervision and permission controls far more important than in the paper-checking story.

    Remember thisComputer use lets software work through screens in an observation-action loop. It is useful when no reliable connector exists, but each action needs checks because interfaces, labels and session state can change.

    Understand it

    Understand it

    Why it matters

    For non-specialists, computer use matters because it changes what an AI-enabled system can do. A chatbot mainly produces text. A system with computer use can operate websites and software directly, which means it can save time on repetitive digital tasks but can also create new operational risks. It may click the wrong control, submit incomplete information, or act in the wrong account if supervision is weak. Leaders and policy teams should therefore ask different questions: What systems may it access? What actions are allowed? What evidence and logs are kept? When must a human approve a step? Where no API exists, computer use may be useful, but convenience should not be mistaken for reliability or safety.

    How computer use completes a task through a screen

    After viewing this, a reader should be able to describe computer use as a repeated observe-act-check loop over a graphical interface, and explain why it needs supervision.

    1. 01

      1. Goal enters

      A person gives a task, such as "download the latest public logistics report and copy the total tonnage into this spreadsheet."

      Draw a rounded box with a document icon and short instruction text. Add a smaller note below: "rules, permissions, limits".

    2. 02

      2. Screen is observed

      The system receives the current page or application state, for example a screenshot or accessibility data describing buttons, fields and labels.

      Draw a monitor icon inside the box. Show callouts to visible items such as button, menu and text field.

    3. 03

      3. Action is chosen

      Based on the goal and the current screen, the system selects the next low-level step, such as click, type, scroll or key press.

      Use a pointer icon and a short action list: click, type, scroll, tab. Keep one item highlighted.

    4. 04

      4. Action happens

      The chosen step is sent to the browser or desktop session. The interface changes, or sometimes fails to change.

      Show a cursor clicking a button and a subtle state change, such as a dialog opening. Add a small branch note: "if blocked or unclear, ask human".

    5. 05

      5. Result is checked

      The system compares the new screen with the goal. It either loops for another step, finishes the task, or stops for review.

      Show a decision-style box with three small outcomes beneath it: continue, done, escalate.

    A left-to-right step diagram with five boxes linked by arrows. Read from the user goal on the left to the final result on the right. Each middle box shows one cycle stage: seeing the screen, choosing an action, doing it, then checking what changed. A small warning badge appears near the last two steps to show where human review may be needed.

    Clicking a step reveals one example action and one possible failure, such as a pop-up covering the target button.

    Describe this without the diagram

    The diagram shows computer use as a five-step sequence. First, a human provides a task and any rules or limits. Second, the system observes the current screen, using a screenshot, accessibility information, or both. Third, it selects one interface action, such as clicking, typing or scrolling. Fourth, that action is executed in the browser or desktop session. Fifth, it checks the new screen state against the goal. If the task is incomplete, it repeats the cycle. If the task is complete, it stops. If the situation is risky or unclear, it asks for human review. The main idea is that computer use is not one command but a loop of observation and action through a user interface.

    With reduced motion: Show the same five boxes as a static horizontal sequence with arrows. Replace any animation with numbered labels and a dashed loop-back arrow from the final checking step to the observation step.

    Computer use means an AI system can work through a graphical interface, such as a website or desktop app, instead of only replying with text or calling a fixed API. The input is usually a goal from a person, plus what the system can currently see on the screen. That screen view may be an image, a structured accessibility tree, or both.

    The system then follows a loop: observe, choose an action, act, and check the result. An action might be moving a pointer, clicking a button, typing into a field, scrolling, or pressing keys. After each action, it looks again to see what changed and whether it is closer to the goal.

    The output is not just an answer. It can be a completed task, such as filling a form, downloading a file, or copying information from one system to another. It may also produce a log of steps, screenshots, or a note asking a human to confirm a risky action.

    Computer use is one form of tool use. It acts through a graphical interface using screen observations and low-level actions. Tool use also includes APIs, other software and devices. Function calling is the separate method of returning a structured request to a defined external function.

    Because it can take actions in real systems, it needs strong limits, monitoring and human oversight. Small screen changes, pop-ups, language differences or missing permissions can cause errors, so people should not treat it as reliable simply because it can click through many steps.

    How “Checking a delivery against a shipping manifest” maps onto the real thing

    In the storyIn the AI system
    the manifest with a goalthe user instruction describing the task
    looking at labels on each palletobserving the screen after each step
    marking the list and moving to the next pallettaking interface actions such as click, type or scroll
    asking the supervisor about damaged or unclear labelshanding uncertain or risky cases back to a human

    A worked example

    An approved assistant is asked to transfer one cleared progress status from an internal results portal to the corresponding row of a draft annual workplan review workbook.

    Starting from: Goal: "Follow the portal link in the selected workbook row, read the cleared progress status, enter the matching status in the workbook and stop before submission."

    1. It observes the selected workbook row, identifies the portal link and opens it in a new tab.
    2. It observes the output record, scrolls until the cleared progress field is visible and reads the displayed status.
    3. It returns to the workbook, opens the status dropdown in the same row and selects the matching option.
    4. It observes the updated row, compares it with the portal entry and reports the action. If the labels do not match exactly, it stops for the reporting officer to decide.

    Result: The draft workbook contains the status shown in the cleared portal entry, together with a short action record. The assistant does not submit the review or treat the transfer as clearance.

    In a United Nations workplace

    At a duty station working in three languages, a reporting officer uses an approved computer-use assistant during the annual workplan review. The assistant opens the internal results portal, locates each output's cleared progress entry and enters the status in the review workbook before close of business.

    This can reduce repetitive clicking where no reliable connector exists. The assistant can scroll, switch tabs and operate dropdown menus, but a changed layout, expired session or similar-looking label can send the action loop in the wrong direction. The reporting officer still checks the workbook against the cleared entries.

    Before you try thisAn unapproved tool must not be given access to the portal, staff credentials or restricted records. If a screen contains personal, confidential or security-sensitive material, the task must not run unless the tool is specifically approved and configured for that material. Submission, clearance and external circulation should remain subject to human confirmation.

    Where you may hear it

    • A vendor demonstration to an ICT governance board
    • A procurement unit meeting on the technical evaluation of an automation tool
    • A country office workplan review meeting
    • A risk review before an enterprise assistant pilot

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • Computer use is just another name for automation.

      Automation is the broader category. A scripted automation may follow fixed rules in one stable system. Computer use is a specific way of acting through a screen interface, often step by step, using observations of what appears on the screen. It is usually more flexible than a hard-coded script, but also more fragile because layout changes, pop-ups or ambiguous labels can break the task.

    • If a system can use a browser like a person, it will be good at any office task.

      Using the same interface as a person does not guarantee reliable performance. Real tasks often involve missing context, ambiguous instructions, permissions, multilingual screens, hidden fields, timing issues and judgement calls. A system may complete simple repetitive steps yet still fail on exceptions or risky actions that need human review.

    • Computer use and tool use are the same thing.

      They overlap but are not identical. Tool use usually means the model calls a defined tool, function or API with known parameters, such as get_weather(city). Computer use acts through visible controls on a screen, like clicking a calendar menu or typing in a search box. APIs are usually cleaner and more reliable; computer use is often chosen when no suitable API or connector exists.

    Quick check

    What most clearly distinguishes computer use from ordinary text generation?

    Go deeper

    Go deeper

    The simple version

    It lets an AI-enabled system operate software the way a person would: by looking at the screen and taking small actions such as clicking, typing and scrolling until the task is done or a human needs to step in.

    The technical version

    Computer use is an agent capability for interacting with graphical user interfaces through an observation-action loop. The observations may be screenshots, DOM elements, accessibility trees or other interface state representations. The action space typically includes pointer movement, clicks, keyboard input, scrolling, focus changes and window or tab switching. A controller or model uses the current goal, recent history and the latest observation to choose the next action, then evaluates the resulting state and repeats. Compared with direct API or function access, this approach is interface-dependent, higher-latency and more error-prone, but it can work across systems that expose no stable programmatic interface. In practice it is often combined with planning, short-term memory, guardrails, logging and human approval checkpoints for high-impact actions.

    Reviewed external resources

    • Primary sourceModel Context ProtocolModel Context Protocol

      This official resource explains the Model Context Protocol and suits technical readers who want to understand one approach for connecting systems and tools.

      Deeperlink checked 2026-07-31

    • EncyclopaediaWikipedia search: Computer useWikipedia

      This search results page helps readers explore different Wikipedia articles related to the phrase and suits readers who want broad starting points rather than one definitive entry.

      search results, not one pagelink checked 2026-07-31

    • ArticleBuilding effective agentsAnthropic

      This article gives a vendor explanation of agents and is useful for readers who want a practical overview of how systems can take multi-step actions.

      Intermediatelink checked 2026-07-31

    • VideoVideo explanations: Computer useYouTube search

      This search results page helps readers compare multiple video explanations of computer use and suits people who learn better by watching demonstrations.

      search results, not one pagelink checked 2026-07-31

    • DiscussionCommunity discussion: Computer useReddit search

      This search results page shows community discussions and examples, which can help readers see common questions and concerns, but it suits readers who can critically assess mixed-quality commentary.

      search results, not one pagelink checked 2026-07-31

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    Computer use lets software work through screens in an observation-action loop. It is useful when no reliable connector exists, but each action needs checks because interfaces, labels and session state can change.