Do role-playing language agents accurately capture character personalities? Towards this question, we propose InCharacter that evaluates their personality fidelity via psychological interviews using 14 scales.

We present a comprehensive survey on role-playing language agents (RPLAs), i.e., specialized AI systems simulating assigned personas. Specifically, we distinguish personas in RPLAs into three progressive tiers - demographic persona, character persona, and individualized persona. Our survey discusses their data sourcing, agent construction, evaluation, applications, risks, limitations and future prospects.

Can Large Language Models substitute humans in making important decisions? Recent research has unveiled the potential of LLMs to role-play assigned personas, mimicking their knowledge and linguistic habits. However, imitative decision-making requires a more nuanced understanding of personas. In this paper, we benchmark the ability of LLMs in persona-driven decision-making. Specifically, we investigate whether LLMs can predict characters’ decisions provided with the preceding stories in high-quality novels. Leveraging character analyses written by literary experts, we construct a dataset LIFECHOICE comprising 1,401 character decision points from 395 books. Then, we conduct comprehensive experiments on LIFECHOICE, with various LLMs and methods for LLM role-playing. The results demonstrate that state-of-the-art LLMs exhibit promising capabilities in this task, yet there is substantial room for improvement. Hence, we further propose the CHARMAP method, which achieves a 6.01% increase in accuracy via persona-based memory retrieval. We will make our datasets and code publicly available.

We propose a novel conversational AI system that enhances researchers’ literature review processes by providing personalized knowledge management, literature recommendations, and query answering through a unified platform.