🔬 Research Summary by Karim Benharrak, a first-year CS PhD student at the University of Texas Austin, where I design, develop, and evaluate interactive AI systems to unlock the collaborative potential of Human-AI interaction in content creation processes.
[Original paper by Karim Benharrak, Tim Zindulka, Florian Lehmann, Hendrik Heuer, and Daniel Buschek]
Overview: People write for people: Writing is inherently social, yet writers often lack real-time feedback from their intended audience. To support writers, this paper introduces the concept of ‘AI Personas for On-Demand Feedback.’ Through our new text editor system, Impressona, writers receive feedback from various perspectives on how they define themselves by creating personas of their target reader groups.
Introduction
In a world where writing is an integral part of our daily lives, the challenge of effectively connecting with our readers remains ever-present. How can writers ensure that their words resonate with their intended audience? This research addresses these pressing questions.
The core focus of this study revolves around the concept of “AI Personas for On-Demand Feedback,” a novel approach aimed at empowering writers to receive real-time, reader-oriented feedback on selected parts of their texts. But what does this mean in practice? Writers define personas representing their target readers, and a Large Language Model (LLM) provides insightful feedback on their writing from these multiple perspectives. We implemented this concept in a system called Impressona, a tool designed to bridge the gap between writers and their readers.
The paper explores the effectiveness of AI personas in enhancing the quality of written content and the writer-reader connection. Through two user studies, we uncover how writers perceive and utilize self-defined AI personas while also gaining insight into the capabilities of contemporary LLMs in supporting this concept. With these findings, the paper demonstrates a powerful new tool for writers and contributes to the broader vision of AI’s role in content creation and its impact on the dynamic relationship between creators and their audience.
Key Insights
Here, we delve into the key findings and implications of our study. By allowing writers to define AI personas that provide on-demand feedback, we open new avenues for enhancing the writing process by supporting writers in tailoring their text to their readers/audience.
Empowering Writers to Shape Feedback
In our study, we introduced the concept of Writer-Defined AI Personas. Our concept enables writers to define custom AI personas during the writing process by defining characteristic properties of their imagined readers. The process involves crafting personas based on attributes like expertise, social relation, valence, and level of involvement. For example, writers can create personas like a “strict professor” or a “friendly colleague” with attributes describing their task, background, and preferred style and content. These personas enable writers to gain feedback on specific portions of their text from perspectives they desire.
Understanding Persona Dimensions
Our user studies uncovered a taxonomy of AI personas for feedback, shedding light on the dimensions that writers consider when creating such AI personas. The most commonly observed attribute was expertise, often linked to professional roles and qualifications (e.g., “Oxford Professor,” “NYTimes Journalist”). Social relations also played a significant role, with personas like “best friend” and “my manager” offering different perspectives. Writers also considered the valence (e.g., “mean reviewer”) and level of involvement (e.g., “my Ph.D. colleague,” “Scientific Writer”), providing a range of feedback perspectives.
Quick Access to Multiple Perspectives Through Feedback From AI Personas
Our study revealed that AI personas for feedback offer substantial benefits to writers. Writers highly value and use feedback from these personas, finding it useful and diverse. Most notably, the personas empower writers to consider multiple perspectives, making them step outside their writing and gain fresh insights. Moreover, writers can access feedback from personas they might not easily reach in reality, offering the convenience of instant feedback.
Persona Definition Hurdles and Feedback Verbosity
While the concept of AI personas for feedback is promising, challenges exist. Writers may find it initially difficult to define personas accurately, which could impact the quality of feedback. The feedback generated by AI personas can sometimes be verbose, repetitive, and unspecific, raising questions about the optimal balance between feedback length and usefulness.
Guarding Against Bias and Stereotypes
Our research addresses the critical concern of bias and stereotypes in contemporary AI systems. It highlights the risk of AI models encoding the dominant view and failing to represent changing social perspectives. This is particularly important when considering attributes like gender, race, ethnicity, and disability status. Moreover, while not a major concern in our study, we observed that feedback from AI personas could influence writers’ attitudes on whether to make changes to their content.
In light of these concerns, it’s important to approach the concept of AI personas with caution. Systems that provide AI personas should warn users about the potential limitations of contemporary AI systems regarding representativity. The flexibility of our system, allowing writers to change personas at any time, offers one way for writers to react to unwanted aspects in the feedback if they recognize those themselves.
In conclusion, our research introduces an innovative approach to enhancing writing feedback by empowering writers to define AI personas. While challenges exist, the potential benefits are evident already in our exploration, with key implications for the design of writing support systems: As LLMs continue to shape the way we interact with technology, the concept of AI personas -that can comment on text- opens exciting possibilities for improving the human writing process with opportunities for reflection that empathize with potential reader perspectives.
Between the lines
Shaping the Future of Writing Support
Beyond the specific system and study, this work calls for a shift in our approach to designing AI-driven writing tools: Instead of designing for human-AI interaction, we should consider a broader socio-technical perspective, that is, designing with the writer-reader relationship in mind. This expanded perspective acknowledges the impact of AI on both writers and readers, providing a more comprehensive approach to design.
One of the key insights is the newfound agency given to writers, allowing them to define and refine personas to meet their specific needs. This questions how personas can evolve, adapting to writers’ changing preferences and requirements. Looking ahead, it challenges us to delve deeper into understanding how AI personas can support writers and facilitate connections between writers and readers, transcending traditional writing and publication processes and tools.