Apple Develops AI to Understand and Interact with App Interfaces
• Apple is developing Ferret-UI, an AI system to understand app screens and interfaces • Ferret-UI is designed to recognize small UI elements like icons and text • It can interact with apps and summarize their purpose and functions • Potential uses include evaluating UIs, improving accessibility, and powering an advanced Siri • Ferret-UI performed better than other multimodal AI systems at understanding app UIs