The Problem with Traditional Approaches
Screens are deeply embedded in how we live, learn, and connect—yet research often reduces digital experience to app-based metrics like time spent per app.
What App-Based Metrics Miss:
📱 What users actually see: An Instagram feed of makeup tutorials vs. cyberbullying is treated the same, despite vastly different effects.
🔄 How content flows across apps: Users jump between apps and browsers for related content, but this cross-platform flow is often overlooked.
🌱 Emergent behaviors: App categories are rigid and predefined. They don’t adapt to the diverse ways users engage with content—or how those behaviors evolve over time.
To address these blind spots, recent research has turned to continuous screen data collection, which captures everything displayed on users' screens.
Despite these advances, there are still no scalable and flexible tools that allow researchers to interact with, explore, and analyze massive, sensitive, and unstructured multimodal data.
Beyond App Names: Capturing Media Content via Screenomics
To design and evaluate Media Content Atlas, we used 1.12 million screens (10,000 screenshots per participant) collected from 112 participants over the course of one month. This data comes from the Human Screenome Project, where a study-specific app captured a screenshot every 5 seconds whenever the screen was on, creating a continuous, moment-by-moment record of digital experiences.
🔎 The Scale Challenge
Manually reviewing this dataset would take 190+ workdays, making AI-augmented exploration and analysis essential.
Figure. A short slice of a Screenome. This 15-minute segment, recorded from a consenting researcher’s real-world smartphone use, illustrates how digital content and media experiences shift even over brief periods of time. Colors indicate different app categories, while the size of each bar represents the amount of time spent on each app category.
