AI4Democracy Series 1: Using AI to Inform Policymaking

Good policymaking requires a multifaceted approach, incorporating diverse tools and processes to address the varied needs and expectations of constituents. The paper by Turan and McKenzie focuses on an LLM-based tool, "Talk to the City" (TttC), developed to facilitate collective decision-making by soliciting, analyzing, and organizing public opinion. This tool has been tested in three distinct applications:

1. Finding Shared Principles within Constituencies: Through large-scale citizen consultations, TttC helps identify common values and priorities.

2. Compiling Shared Experiences in Community Organizing: The tool aggregates and synthesizes the experiences of community members, providing a cohesive overview.

3. Action-Oriented Decision Making in Decentralized Governance: TttC supports decision-making processes in decentralized governance structures by providing actionable insights from diverse inputs.

Capabilities and Benefits of LLM Tools

LLMs, when applied to democratic decision-making, offer significant advantages:

• Processing Large Volumes of Qualitative Inputs: LLMs can handle extensive qualitative data, summarizing discussions and identifying overarching themes with high accuracy.
• Producing Aggregate Descriptions in Natural Language: The ability to generate clear, comprehensible summaries from complex data makes these tools invaluable for communicating nuanced topics.
• Facilitating Understanding of Constituents' Needs: By organizing public input, LLM tools help leaders gain a better understanding of their constituents' needs and priorities.

Case Studies and Tool Efficacy

The paper presents case studies using TttC, demonstrating its effectiveness in improving collective deliberation and decision-making. Key functionalities include:

• Aggregating Responses and Clustering Ideas: TttC identifies common themes and divergences within a population’s opinions.
• Interactive Interface for Exploration: The tool provides an interactive platform for exploring the diversity of opinions at both individual and group scales, revealing complexity, common ground, and polarization.

Best Practices for LLM-Assisted Deliberation

Based on the analysis of three case studies, the authors advance the following best practices:

1. Seek Rich, Substantial Datasets: The quality of LLM-generated reports improves with the size and richness of the datasets analyzed. Collecting substantial data ensures more interesting and comprehensive summaries.Consider the trade-offs between the volume of ideas and report brevity when collecting data.

2. Iterate on Prompts for Ideal Topic Hierarchies: Initial topic hierarchies may not always clearly represent the overall themes. Iterating on prompts can refine the clarity and relevance of the topics identified. Amend prompts to merge or rephrase overlapping topics to generate clearer and more representative reports.

3. Inspect Results for Conflation, Not Hallucination: While hallucinations are a known risk, conflation of similar ideas is a more significant issue. Carefully review reports to identify and correct any miscategorized claims.

4. Supplement LLM Analysis with Contextual Materials: For topics requiring significant background knowledge, supplement LLM-generated reports with additional context and explanations of jargon or terms of art. This approach ensures that reports are accessible and informative for readers unfamiliar with the specific context.

5. Use Audio and Video Content When Possible: Incorporating audio and video inputs enhances the impact of LLM summaries. Video interviews, in particular, provide a powerful, humanizing element that text alone cannot convey. Combining summary reports with multimedia content offers a richer, more comprehensive understanding of the discussed topics.

The integration of AI, particularly LLMs, into the policymaking process offers promising avenues for enhancing democratic decision-making. Tools like Talk to the City demonstrate the potential to support collective agency and improve AI governance and safety. By adopting best practices and continuously refining these tools, policymakers can harness the full potential of AI to create more informed, responsive, and effective governance structures.

For those interested in reading the report: AI4Democracy