AGI Applications are moving from hype to measurable gains, with studies from 2023 to May 2025 showing faster task completion, higher accuracy, and improved generalization across real desktop software. In Office Word tasks, an API-first agent framework cut task time by 65–70% and reduced cognitive workload by 38–53%, while reaching 97–98% human-comparable accuracy [2]. GUI-grounded agents now complete workflows across nine popular Windows applications using dual-agent control with GPT-Vision [3]. A 2025 preprint reports sustained exploration and stronger generalization using curiosity-driven training in an open GUI world [1]. HCI guidance urges new metrics and safeguards to integrate these agents safely [4]. Open-source Agent UI stacks from 2023–2025 enable rapid deployment [5].

Key Takeaways

– Shows API-first agents cut task time 65–70% and cognitive workload 38–53% in Word tasks, reaching 97–98% human-comparable accuracy. – Reveals GUI agents like UFO complete tasks across nine Windows apps using dual-agent control and GPT-Vision, with superior completion versus baselines. – Demonstrates ScreenExplorer’s May 25, 2025 results: GRPO plus curiosity sustains exploration and generalizes to novel apps in dynamic, open interfaces. – Indicates the 2023 UIST HCI paper urges new evaluation metrics, safeguards, and timelines so AGI Applications integrate safely into routine, high-stakes workflows. – Suggests 2023–2025 open-source Agent UI projects add deployable chat UIs, multi-app tooling, streaming, and model-stack integration for application-facing agents.

Why AGI Applications are shifting to API-first control

A central finding of the 2024 AXIS framework is that LLM-based agents should prioritize application APIs ahead of brittle pixel clicking and keystroke sequences [2]. By calling structured functions, the agent bypasses visual ambiguity and reduces error propagation seen in pure GUI manipulation [2]. In controlled Office Word evaluations, AXIS cut task time 65–70%, lowered cognitive workload by 38–53% on NASA-TLX-style measures, and reached 97–98% accuracy relative to human performance [2]. The authors argue API-first design is critical to scale LLM agents to practical, application-level AGI, minimizing latency and compounding misclicks [2].

These results also clarify where UI automation remains essential. Many apps lack comprehensive APIs, so hybrid strategies maintain GUI fallbacks while preferring APIs when available [2]. This suggests product teams should expand API coverage to unlock agent speed and reliability gains [2]. The data makes a straightforward business case: where thorough APIs exist, agents become faster, more accurate, and less cognitively taxing to supervise, enabling safer semi-autonomy [2].

GUI-grounded agents prove real-world breadth across nine apps

Purely API-first is not always possible. Microsoft-affiliated researchers showed UFO, a UI-focused agent, can observe and act across the Windows OS using a dual-agent architecture with GPT-Vision [3]. Tested on nine popular desktop applications, UFO achieved superior task completion rates versus baselines, demonstrating that vision-enabled agents can navigate varied toolbars, dialogs, and transient UI states [3]. The authors open-sourced the code, positioning GUI-focused agents as a viable bridge toward application-level AGI where APIs are incomplete or absent [3].

The nine-app breadth matters for enterprise relevance. Real estates of legacy software and proprietary tools often resist API exposure, forcing agents to “see” and manipulate interfaces like people do [3]. In such contexts, robust screenshot understanding, control localization, and error recovery are prerequisites for dependable automation, and the reported results indicate progress on each dimension [3].

Training breakthroughs push open-ended exploration

Exploration is a bottleneck for agents learning new software. On May 25, 2025, the ScreenExplorer preprint introduced a vision-language model trained with Group Relative Policy Optimization and a curiosity reward to sustain exploration in open, dynamic GUI environments [1]. The experiments reported stronger generalization across unseen applications and continued exploration without early collapse, a common failure mode in sparse-reward UI tasks [1]. The authors argue this combination of RL fine-tuning and intrinsic motivation helps scale agents toward AGI-capable behavior on ever-changing interfaces [1].

Two implications follow. First, curiosity-based rewards can counteract local maxima, prompting agents to discover hidden menus and context-specific affordances [1]. Second, group-relative optimization stabilizes policy improvement by comparing against peers, potentially reducing regressions during training on diverse UI states [1]. Both mechanisms address the churn of real-world GUIs, where updates frequently invalidate brittle action scripts [1].

HCI readiness and safeguards for autonomous AGI Applications

As capabilities accelerate, the HCI community has urged proactive design of metrics and safeguards. A 2023 UIST adjunct paper questions whether current interfaces, evaluation methodologies, and governance are ready for AGI-era systems [4]. The authors recommend new interaction models, risk-informed evaluation criteria, and adoption timelines that match autonomy growth, to avoid misalignment between agent competence and user control [4]. They stress the need for guardrails that surface intent, highlight irreversible actions, and allow seamless interruption during high-stakes operations [4].

For enterprises piloting AGI Applications, this translates into dashboard-level observability, clear affordances for “approve/undo,” and transparent logs for audit and compliance [4]. The paper’s call to redesign around autonomous agents suggests organizations should treat agent UX as a first-class surface, not an afterthought layered atop legacy flows [4]. Measurable UX gains must be balanced with safety-by-design patterns that anticipate failure modes [4].

Open-source UI stacks accelerate deployment of AGI Applications

Beyond research, developers now have production-grade scaffolding to ship agent experiences. Open-source Agent UI projects active from 2023 through 2025 provide modern chat and multi-app interfaces built with Next.js and Tailwind, with streaming updates and connectors for local AgentOS and model stacks [5]. The repositories demonstrate practical tooling to orchestrate prompts, route tool calls, and present agent state in ways that end users can supervise and refine [5]. This community infrastructure lowers the barrier to put AGI Applications in users’ hands and iterate quickly on UX [5].

In practice, these stacks shorten the path from model capability to product integration. Teams can plug in API-first policies where supported, layer GUI control for gaps, and expose consistent feedback channels for human-in-the-loop corrections [5]. The cumulative effect is faster experimentation on agent autonomy levels while maintaining usable, debuggable front-ends [5].

Unifying metrics: speed, accuracy, cognitive load, and generalization

Across studies, several metrics recur: time-to-completion, task accuracy, cognitive workload, and generalization to unseen interfaces [2]. API-first control moved these dials most visibly in Word tasks, but the pattern establishes a template for benchmarking other productivity suites and line-of-business apps [2]. GUI agents add breadth across multi-app workflows, which must be quantified via standardized completion and recovery metrics in noisy environments [3]. Meanwhile, exploration research targets long-horizon generalization and resistance to brittle scripts, calling for measures that capture sustained discovery behavior [1].

HCI guidance complements the technical frame. Risk-weighted evaluations, visibility of agent intent, and intervention affordances should be codified into user studies and acceptance criteria, not left as ad hoc checks [4]. A shared metric set—spanning performance, workload, and safety—would allow organizations to compare agents fairly and track progress as product landscapes change [4].

Design trade-offs: API coverage versus GUI robustness

The data suggest AGI Applications should default to APIs for speed and accuracy, but enterprise teams must inventory API coverage versus critical tasks to quantify return on investment [2]. Where API gaps exist, vision-driven GUI control must be reliable under UI churn and latency, favoring architectures that separate perception from action and log every step for replay and diagnosis [3]. Curiosity-driven training could further bolster GUI robustness by encouraging exploration that uncovers edge-case controls and states [1].

In this hybrid world, product decisions hinge on measurable trade-offs. If adding a missing API unlocks 65–70% cycle-time savings for a frequent task, the integration likely pays back quickly [2]. If not, investing in GUI policies with solid completion and recovery rates may be the pragmatic path while APIs evolve [3]. HCI-derived safeguards remain non-negotiable in either route [4].

Governance and human oversight in deployment

Autonomy is not a switch; it is a gradient set by risk and evidence. The UIST paper recommends aligning autonomy with clear milestones, ensuring operators can supervise, approve, and reverse critical actions without friction [4]. For AGI Applications, that means tiered permissions, change review for destructive operations, and transparent logs to support post-incident analysis and regulatory audits [4]. Open-source UI layers already expose many of these controls, which teams can adapt to domain-specific policies [5].

The reported 97–98% accuracy in Word tasks is impressive but still implies non-trivial error rates at scale, reinforcing the need for well-designed human-in-the-loop interactions [2]. Clear status messaging, diff previews before document changes, and one-click rollbacks exemplify how UX can internalize safety without stalling productivity gains [4].

How research trajectories are converging

A plausible near-term convergence combines API-first action selection, GUI fallback via robust vision, and curiosity-enhanced exploration to adapt to UI changes [2]. UFO-like perception and control provide breadth when APIs are incomplete, while ScreenExplorer-style training sustains learning in unfamiliar territories [3]. When APIs are available, AXIS-like policies drive higher speed and accuracy, delivering the headline efficiency wins [2]. Open-source Agent UIs then translate these capabilities into observable, steerable experiences for end users [5]. HCI frameworks guide evaluation and guardrail implementation across this stack [4].

Together, these strands aim squarely at application-level AGI: agents that can reliably operate existing software ecosystems, learn new ones with minimal supervision, and expose controls that keep humans confidently in charge [1].

What enterprises should do now

First, audit API coverage for target applications and map tasks where a 65–70% time reduction would unlock significant throughput gains [2]. Second, pilot GUI agents in multi-app workflows that lack APIs, measuring completion rates and recovery paths under realistic UI changes [3]. Third, test exploration-driven training on high-churn software to validate generalization before broad rollout [1]. Fourth, adopt HCI recommendations—risk-scored evaluations, intent transparency, and interruptibility—into acceptance gates for any autonomous feature [4]. Finally, stand up open-source Agent UIs to accelerate iteration and centralize supervision [5].

By grounding programs in these validated levers, organizations can move beyond prototypes and capture measurable ROI while containing operational risk [2].

Open questions for 2026 and beyond

Key open questions include how well exploration advances transfer across domains, how quickly API coverage can expand in legacy suites, and what standardized metrics regulators will expect for agent safety [1]. Another is how to measure cognitive workload for supervisors at scale, ensuring that oversight does not reintroduce friction as autonomy rises [2]. The open-source ecosystem will likely answer part of this by hardening observability patterns and governance components over continued 2023–2025 development cycles [5]. Expect more cross-benchmark validation on nine-plus-app suites and beyond [3].

Sources:

[1] arXiv (preprint) – ScreenExplorer: Training a Vision-Language Model for Diverse Exploration in Open GUI World: https://arxiv.org/abs/2505.19095

[2] arXiv (preprint) – Turn Every Application into an Agent: Towards Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents: https://arxiv.org/abs/2409.17140 [3] arXiv / Microsoft GitHub – UFO: A UI-Focused Agent for Windows OS Interaction: https://arxiv.org/abs/2402.07939

[4] ACM UIST Adjunct Proceedings – AGI is Coming… Is HCI Ready?: https://dl.acm.org/doi/10.1145/3586182.3624510 [5] GitHub (big-AGI, agno-agi/agent-ui) – Screen and Agent UI projects (big-AGI / Agent UI) — open-source agent-oriented UIs and frameworks: https://github.com/agno-agi/agent-ui

Image generated by DALL-E 3