Assessing the quality of political online discussions is crucial for understanding and fostering democratic discourse, yet automating this process remains challenging. While research has identified various indicators to evaluate the deliberative quality of online discussions, existing approaches often focus on isolated aspects rather than a holistic measure of deliberative quality. In addition, most approaches are based on manual content analysis. With advancements in deep learning, it is now possible to develop AI-driven methods that enhance transparency and reliability in the analysis of online deliberation.
We introduce AQuA, an additive score that quantifies deliberative quality based on multiple indices for each discussion post. Unlike singular scores, AQuA retains detailed information on different deliberative aspects, ensuring greater interpretability. We develop adapter models for 20 deliberative indices and use correlation coefficients between expert annotations and non-expert perceptions to weigh these indices into a unified deliberative score. Our results demonstrate that AQuA can be efficiently computed from pre-trained adapters and generalizes well to unseen datasets. Moreover, our comparison of expert and non-expert annotations provides empirical support for theoretical findings in social science research.
This presentation will be relevant for data journalists, data scientists, and researchers from communication science as well as computational social science working on AI applications to analyze online deliberation. We will discuss the methodological foundations of AQuA, demonstrate its practical applications in assessing the deliberative quality of online discussions, and show its potential for integrating AI-driven analysis of online deliberation into journalistic and academic workflows.