As on-device artificial intelligence evolves from simply "answering questions" to maintaining a long-term understanding of users and proactively executing tasks, memory capabilities are becoming the foundational pillar for delivering personalized and proactive AI services. In a significant industry move, OPPO, in collaboration with the OpenKG community and academic experts from Zhejiang University, Tongji University, and Southeast University, has launched MobileMem, the first benchmark specifically designed to evaluate the memory capabilities of on-device AI systems.
By creating a unified and open evaluation framework, MobileMem provides a standardized "ruler" that allows different technical approaches to be compared and iterated upon under consistent criteria. This initiative accelerates the development of on-device AI memory technology, extending OPPO's commitment from its own product development to fostering broader industry-wide collaboration and ecosystem growth.
Establishing a Standard Measure for On-Device AI Memory
While significant research progress has been made in evaluating long-term memory and long-horizon agent capabilities, a systematic assessment tailored to the long-term personalized and multimodal scenarios found on mobile devices has been lacking. Unlike cloud-based agents, on-device AI must continuously accumulate, update, and leverage long-term user memory while operating under constraints of limited computing power, storage, and privacy protection. These unique boundaries and optimization goals necessitate a distinct evaluation approach. The absence of a unified, open benchmark has hindered developers from conducting effective A/B testing and performance regression analysis, and has made it difficult for researchers to objectively compare different solutions under consistent tasks and evaluation protocols, thereby impeding technological iteration and ecosystem development.
In response, MobileMem was developed based on cutting-edge research and practical on-device scenario experience. The benchmark's dataset includes historical context totaling millions of tokens, simulating the long-term usage patterns of real users. Building upon prior work in long-term memory, long context, and multimodal agents, MobileMem focuses on the real-world usage and long-term personalization needs of on-device AI, primarily through three core technical dimensions.
The first dimension is long-term event modeling and retrieval, which covers the storage, updating, and retrieval of events across time, applications, and modalities. The second is cross-modal association and reasoning, which involves understanding and reasoning over relationships between text and image information, including temporal reasoning, relational reasoning, and question answering. The third is continuous evolution and personalized adaptation, which addresses the ongoing updating of user preferences, social relationships, and event states as they change over time.
For example, in a "personal life companion" scenario, a family planning their annual vacation might ask: "Can you help me plan a trip similar to the one we enjoyed last year?" Instead of generating generic travel recommendations, the assistant leverages its long-term memory to propose a personalized itinerary that reflects the user's historical preferences, travel habits, and spending patterns, while also avoiding previously noted inconveniences. This example highlights MobileMem's capacity to manage complex personal memory tasks, integrating long-term preference modeling, temporal information aggregation, multimodal reasoning, memory association, and personalized decision-making.
MobileMem: From Contributing Data to Building an Ecosystem
MobileMem provides a unified evaluation reference for on-device AI memory research, effectively connecting research, development, and real-world products. Its test framework is compatible with various agents and on-device intelligence systems, covering modules for task understanding, memory retrieval, search, and execution. This enables developers to test their solutions against a standard and continuously monitor performance to support rapid product iteration.
A senior OPPO executive stated that long-term memory is a vital foundation for achieving personalized intelligent experiences on devices. The company emphasized that MobileMem is not just an evaluation benchmark, but a critical piece of industry infrastructure designed to accelerate the implementation of memory technology in on-device AI. By making this "ruler" open, OPPO aims to collaborate with the industry in exploring the future of on-device AI. The sentiment was echoed by partners and academics, who noted that MobileMem establishes a much-needed reference system for unified evaluation, enabling different technical routes to be compared and iterated upon under a consistent task framework and evaluation metrics. This is seen as a critical foundation for fostering open research collaboration between industry and academia in the field of on-device intelligent memory.
Looking ahead, the MobileMem 1.0 technical white paper is now publicly available, with plans to progressively open-source the evaluation datasets, task sets, and related tools. The initiative welcomes participation from both industry and academia to continuously refine the evaluation system and drive innovation and ecosystem development in on-device AI memory technology.