Tuning LLM-Based Advisors for the Common Good: The Case for Direct Preference Optimization

Large Language Models (LLMs) are often deployed as advisors to consumers, e.g., recommending purchases, and to managers, e.g., suggesting new hires. In fact, LLMs provide advice based on cost or convenience, overlooking broader societal impacts (e.g., carbon footprint when recommending products to a potential customer, or fairness when recommending a new hire to a company). To align LLMs' advice with societal goals like environmental sustainability and gender parity, tuning strategies must integrate the notion of common good. We discuss why Direct Alignment tuning could be preferable to classic Reinforcement Learning from Human Feedback to achieve this integration. Then, we describe and compare two approaches to Direct Preference Optimization: (1) exposing the model tuning examples taken from recommendations and regulations, and (2) mythopoiesis, i.e., model tuning based on synthetic ``legends'', fictional success stories of regulatory compliance (also generated by LLMs). We present a pipeline to evaluate legends' effectiveness in reducing bias and fostering compliance. Our preliminary results suggest that legend-based tuning may enhance engagement and generalization, while direct exposure ensures factual accuracy but risks rigidity.