鱼皮AI导航
鱼皮AI导航
  • 主页
  • 社区
  • AI工具
  • AI知识库
  • AI提示词
  • MCP
  • Skills
  • AI学习资源
  • AI资讯
  • GitHub开源
登录/注册

computer-use-agents

实现类人化计算机交互能力,通过视觉感知屏幕内容、推理操作意图并执行鼠标点击、键盘输入等动作,支持在隔离沙箱环境中安全运行桌面自动化任务,适用于需要GUI层面控制的智能体开发场景。

效率工具数据与分析软件开发
访问 GitHub
文档Skills.MD
name
computer-use-agents
description
Build AI agents that interact with computers like humans do - viewing screens, moving cursors, clicking buttons, and typing text. Covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives. Critical focus on sandboxing, security, and handling the unique challenges of vision-based control. Use when: computer use, desktop automation agent, screen control AI, vision-based agent, GUI automation.
source
vibeship-spawner-skills (Apache 2.0)

Computer Use Agents

Patterns

Perception-Reasoning-Action Loop

The fundamental architecture of computer use agents: observe screen, reason about next action, execute action, repeat. This loop integrates vision models with action execution through an iterative pipeline.

Key components:

  1. PERCEPTION: Screenshot captures current screen state
  2. REASONING: Vision-language model analyzes and plans
  3. ACTION: Execute mouse/keyboard operations
  4. FEEDBACK: Observe result, continue or correct

Critical insight: Vision agents are completely still during "thinking" phase (1-5 seconds), creating a detectable pause pattern.

When to use: ['Building any computer use agent from scratch', 'Integrating vision models with desktop control', 'Understanding agent behavior patterns']

▼
python
复制代码
from anthropic import Anthropic from PIL import Image import base64 import pyautogui import time class ComputerUseAgent: """ Perception-Reasoning-Action loop implementation. Based on Anthropic Computer Use patterns. """ def __init__(self, client: Anthropic, model: str = "claude-sonnet-4-20250514"): self.client = client self.model = model self.max_steps = 50 # Prevent runaway loops self.action_delay = 0.5 # Seconds between actions def capture_screenshot(self) -> str: """Capture screen and return base64 encoded image.""" screenshot = pyautogui.screenshot() # Resize for token efficiency (1280x800 is good balance) screenshot = screenshot.resize((1280, 800), Image.LANCZOS) import io buffer = io.BytesIO() screenshot.save(buffer, format="PNG") return base64.b64encode(buffer.getvalue()).decode() def execute_action(self, action: dict) -> dict: """Execute mouse/keyboard action on the computer.""" action_type = action.get("type") if action_type == "click": x, y = action["x"], action["y"] button = action.get("button", "left") pyautogui.click(x, y, button=button) return {"success": True, "action": f"clicked at ({x}, {y})"} elif action_type == "type": text = action["text"] pyautogui.typewrite(text, interval=0.02) return {"success": True, "action": f"typed {len(text)} chars"} elif action_type == "key": key = action["key"] pyautogui.press(key) return {"success": True, "action": f"pressed {key}"} elif action_type == "scroll": direction = action.get("direction", "down") amount = action.get("amount", 3) scroll = -amount if direction == "down" else amount pyautogui.scroll(scroll) return {"success": True, "action": f"scrolled {dir

Sandboxed Environment Pattern

Computer use agents MUST run in isolated, sandboxed environments. Never give agents direct access to your main system - the security risks are too high. Use Docker containers with virtual desktops.

Key isolation requirements:

  1. NETWORK: Restrict to necessary endpoints only
  2. FILESYSTEM: Read-only or scoped to temp directories
  3. CREDENTIALS: No access to host credentials
  4. SYSCALLS: Filter dangerous system calls
  5. RESOURCES: Limit CPU, memory, time

The goal is "blast radius minimization" - if the agent goes wrong, damage is contained to the sandbox.

When to use: ['Deploying any computer use agent', 'Testing agent behavior safely', 'Running untrusted automation tasks']

▼
python
复制代码
# Dockerfile for sandboxed computer use environment # Based on Anthropic's reference implementation pattern FROM ubuntu:22.04 # Install desktop environment RUN apt-get update && apt-get install -y \ xvfb \ x11vnc \ fluxbox \ xterm \ firefox \ python3 \ python3-pip \ supervisor # Security: Create non-root user RUN useradd -m -s /bin/bash agent && \ mkdir -p /home/agent/.vnc # Install Python dependencies COPY requirements.txt /tmp/ RUN pip3 install -r /tmp/requirements.txt # Security: Drop capabilities RUN apt-get install -y --no-install-recommends libcap2-bin && \ setcap -r /usr/bin/python3 || true # Copy agent code COPY --chown=agent:agent . /app WORKDIR /app # Supervisor config for virtual display + VNC COPY supervisord.conf /etc/supervisor/conf.d/ # Expose VNC port only (not desktop directly) EXPOSE 5900 # Run as non-root USER agent CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"] --- # docker-compose.yml with security constraints version: '3.8' services: computer-use-agent: build: . ports: - "5900:5900" # VNC for observation - "8080:8080" # API for control # Security constraints security_opt: - no-new-privileges:true - seccomp:seccomp-profile.json # Resource limits deploy: resources: limits: cpus: '2' memory: 4G reservations: cpus: '0.5' memory: 1G # Network isolation networks: - agent-network # No access to host filesystem volumes: - agent-tmp:/tmp # Read-only root filesystem read_only: true tmpfs: - /run - /var/run # Environment environment: - DISPLAY=:99 - NO_PROXY=localhost networks: agent-network: driver: bridge internal: true # No internet by default volumes: agent-tmp: --- # Python wrapper with additional runtime sandboxing import subprocess import os from dataclasses im

Anthropic Computer Use Implementation

Official implementation pattern using Claude's computer use capability. Claude 3.5 Sonnet was the first frontier model to offer computer use. Claude Opus 4.5 is now the "best model in the world for computer use."

Key capabilities:

  • screenshot: Capture current screen state
  • mouse: Click, move, drag operations
  • keyboard: Type text, press keys
  • bash: Run shell commands
  • text_editor: View and edit files

Tool versions:

  • computer_20251124 (Opus 4.5): Adds zoom action for detailed inspection
  • computer_20250124 (All other models): Standard capabilities

Critical limitation: "Some UI elements (like dropdowns and scrollbars) might be tricky for Claude to manipulate" - Anthropic docs

When to use: ['Building production computer use agents', 'Need highest quality vision understanding', 'Full desktop control (not just browser)']

▼
python
复制代码
from anthropic import Anthropic from anthropic.types.beta import ( BetaToolComputerUse20241022, BetaToolBash20241022, BetaToolTextEditor20241022, ) import subprocess import base64 from PIL import Image import io class AnthropicComputerUse: """ Official Anthropic Computer Use implementation. Requires: - Docker container with virtual display - VNC for viewing agent actions - Proper tool implementations """ def __init__(self): self.client = Anthropic() self.model = "claude-sonnet-4-20250514" # Best for computer use self.screen_size = (1280, 800) def get_tools(self) -> list: """Define computer use tools.""" return [ BetaToolComputerUse20241022( type="computer_20241022", name="computer", display_width_px=self.screen_size[0], display_height_px=self.screen_size[1], ), BetaToolBash20241022( type="bash_20241022", name="bash", ), BetaToolTextEditor20241022( type="text_editor_20241022", name="str_replace_editor", ), ] def execute_tool(self, name: str, input: dict) -> dict: """Execute a tool and return result.""" if name == "computer": return self._handle_computer_action(input) elif name == "bash": return self._handle_bash(input) elif name == "str_replace_editor": return self._handle_editor(input) else: return {"error": f"Unknown tool: {name}"} def _handle_computer_action(self, input: dict) -> dict: """Handle computer control actions.""" action = input.get("action") if action == "screenshot": # Capture via xdotool/scrot subprocess.run(["scrot", "/tmp/screenshot.png"]) with open("/tmp/screenshot.png", "rb") as f:

⚠️ Sharp Edges

IssueSeveritySolution
Issuecritical## Defense in depth - no single solution works
Issuemedium## Add human-like variance to actions
Issuehigh## Use keyboard alternatives when possible
Issuemedium## Accept the tradeoff
Issuehigh## Implement context management
Issuehigh## Monitor and limit costs
Issuecritical## ALWAYS use sandboxing
效率工具
数据与分析
软件开发

快捷安装

复制到终端去安装 Skill

npxskillsaddsickn33/antigravity-awesome-skills--skill"computer-use-agents"
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
站长:程序员鱼皮 老鱼简历 商务合作 增值电信业务经营许可证
2026 鱼皮AI导航 | 沪ICP备19026706号-6
🔥热门工具
朱雀AI检测朱雀AI检测
朱雀AI检测助手朱雀AI检测助手
LovartLovart
老鱼简历老鱼简历
WritePass官网WritePass官网
面多多面多多
CursorCursor
星月写作星月写作
朱雀AI检测助手朱雀AI检测助手
零感AI零感AI
Skills 推荐
debug
debug 是一个系统化定位软件缺陷根因的技能,适用于用户报告 'fix'、'broken'、'error' 等问题或测试意外失败时。它通过可验证的假设、最小复现、因果链分析和回归测试,确保在修复前准确定位根本原因。
retrieving-developer-knowledge
retrieving-developer-knowledge 用于搜索、获取和整合 Google 官方开发者文档,覆盖 Google Cloud、AI/Gemini、Android、Chrome、Web、Flutter、Go、Firebase 等平台,适用于查询 gcloud CLI 命令、API 语法、IAM 权限或架构对比等场景。
diff-intake
diff-intake 读取 PR 或分支差异,生成结构化 YAML 变更摘要,用于下游 analytics instrumentation 技能。适用于用户提交 PR 链接、分支对比或原始 diff 后需理解变更内容、评估分析覆盖或启动埋点流程的场景。
webapp-testing
提供本地 Web 应用的端到端测试能力,支持页面导航、元素交互、表单填写、动态等待、断言验证、截图捕获、浏览器日志收集及网络请求拦截,适用于前端功能验证与 UI 行为调试。
definition-of-done-generator
自动生成项目完成标准的规范定义与配置文件,提供符合企业级工作流要求的最佳实践指导,输出可直接投入生产环境的代码和验证结果,适用于需要标准化流程审批、合规审查及跨系统集成的复杂项目管理场景。