Elman, Dive into the Latest Advancements in Reinforcement Learning from Human Feedback!

This newsletter explores the latest breakthroughs in Reinforcement Learning from Human Feedback (RLHF), a rapidly evolving field revolutionizing how we align Artificial Intelligence (AI) systems with human preferences. We will delve into three cutting-edge research papers that showcase the power of integrating human guidance into the learning process, enabling the development of more robust, reliable, and aligned AI agents. From multi-agent collaboration to preference tuning across diverse modalities, these papers highlight the transformative potential of RLHF across various AI applications.

HARP: Empowering Non-Experts to Guide Multi-Agent Collaboration

HARP: Human-Assisted Regrouping with Permutation Invariant Critic for Multi-Agent Reinforcement Learning by Huawen Hu, Enze Shi, Chenxi Yue, Shuocun Yang, Zihao Wu, Yiwei Li, Tianyang Zhong, Tuo Zhang, Tianming Liu, Shu Zhang https://arxiv.org/abs/2409.11741

Caption: This image illustrates the two key phases of HARP: Training and Deployment. During training, agents learn to form groups and take actions, receiving feedback from the environment. In deployment, agents can request human assistance to regroup when facing unfamiliar situations, leading to improved performance.

Human-in-the-loop reinforcement learning (HITL-RL) holds immense potential for enhancing multi-agent systems, but existing methods primarily focus on single-agent scenarios. This paper introduces HARP (Human-Assisted Regrouping with Permutation Invariant Critic), a novel framework designed to seamlessly integrate non-expert human guidance into multi-agent reinforcement learning, specifically for group-oriented tasks.

A key innovation of HARP lies in its dynamic group adjustment mechanism. During training, agents autonomously form and refine groupings based on learned Q-values, promoting efficient collaboration. However, recognizing that fixed strategies may not suffice in dynamic deployment environments, HARP empowers agents to actively seek human assistance when group performance falters. This assistance comes in the form of regrouping suggestions, which are evaluated and refined using a novel Permutation Invariant Group Critic (PIGC). This critic addresses the permutation non-invariance problem inherent in traditional methods, ensuring accurate evaluation of human-proposed groupings regardless of agent order. The PIGC leverages a graph representation of the multi-agent environment, where agents are nodes and their interactions are edges. It utilizes a graph convolutional network to compute the Q-value of each group, ensuring permutation invariance.

Evaluations were conducted across six StarCraft II maps with varying difficulty levels. The results demonstrate HARP's effectiveness in leveraging limited human guidance to significantly enhance agent performance. Notably, HARP achieved a perfect 100% win rate across all maps, highlighting its robustness. In contrast, baseline methods, including VAST and GACG, struggled, particularly in challenging scenarios. For instance, on the 5m_vs_6m map, these methods achieved win rates of only around 50% to 70%, while HARP consistently maintained a 100% win rate. This underscores the value of HARP's human-in-the-loop approach in complex multi-agent settings.

The success of HARP can be attributed to its ability to effectively incorporate human intuition into the grouping process. The framework allows non-expert humans to provide valuable input without requiring constant involvement. This is particularly beneficial in real-world scenarios where continuous expert guidance is often infeasible. Moreover, HARP's ability to adapt to dynamic environments and learn from human feedback makes it a promising approach for a wide range of multi-agent applications.

Preference Tuning with Human Feedback: A New Era in Aligning Generative Models

Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey by Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D. Yao, Shi-Xiong Zhang, Sambit Sahu https://arxiv.org/abs/2409.11564

Caption: The image illustrates four prominent architectures for preference tuning in generative models: RLHF (using PPO/REINFORCE), Online DPO, SFT-like, and DPO. These methods leverage human feedback to align model outputs with human preferences, addressing issues like bias and lack of diversity in generated content. Each architecture showcases the flow of information between human preference input, model components like reward models and policy models, and the ultimate generation of outputs.

Deep generative models, despite their remarkable capabilities, often generate outputs that are misaligned with human preferences, exhibiting issues like hallucinations, bias, and toxicity. This comprehensive survey delves into the burgeoning field of preference tuning, a crucial process for aligning these models with human feedback across various modalities, including language, speech, and vision.

The paper meticulously categorizes preference tuning methods based on several key aspects, including sampling (online vs. offline), modality (text, speech, vision, etc.), language (English, non-English, multilingual), and reward granularity (sample vs. token). It provides an in-depth analysis of prominent approaches, including Reinforcement Learning Human Feedback (RLHF), Directed Preference Optimization (DPO), and SFT-like methods. RLHF, often implemented with Proximal Policy Optimization (PPO) or REINFORCE, leverages human feedback to train a reward model and subsequently optimize the policy model using reinforcement learning. DPO, on the other hand, bypasses the reward modeling stage by directly optimizing a preference-based objective:

LDPO (πθ; πref) := – E(x,yw,y1)~D log σ [logo (Breg log (πθ (Yw | X) / Tref (Yw | X)) - Breg log (πθ (Yι | x) / Tref (yl|x)) )].

The survey also explores numerous DPO variants, each addressing specific limitations of the original DPO, such as overfitting, sensitivity to distribution shifts, and lack of diversity in capturing human preferences.

The paper further delves into applications of preference tuning across diverse domains, including instruction-tuned large language models (LLMs), text-to-image generation, and text-to-speech synthesis. It also discusses evaluation metrics and pipelines, highlighting the use of LLMs as judges for automatic evaluation. Notably, the survey underscores the challenges in comparing different preference tuning methods due to variations in hyperparameters, baselines, and evaluation biases.

Finally, the survey identifies promising research directions, emphasizing the need for further exploration in areas like multilinguality, multi-modality, speech applications, and unlearning. It also highlights the importance of developing comprehensive benchmarks and gaining a deeper mechanistic understanding of preference tuning methods to enhance their effectiveness and reliability. This survey serves as a valuable resource for researchers and practitioners alike, fostering a deeper understanding of preference tuning and its potential to shape the future of generative models.

From Lists to Emojis: How Format Bias Affects Model Alignment

From Lists to Emojis: How Format Bias Affects Model Alignment by Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, Tong Zhang https://arxiv.org/abs/2409.11704

Caption: This image shows the impact of format bias on the generation of bold and list patterns in a best-of-n sampling experiment. The x-axis represents the ratio of generated responses containing bold patterns, while the y-axis represents the ratio of responses containing list patterns. As the number of candidate responses increases, models trained on datasets with format biases (e.g., "+Bold Pattern Dataset", "+List Pattern Dataset", "+Bold and List") exhibit a stronger preference for generating those patterns compared to the baseline model or a model trained on a less biased dataset ("Llama3-8B-it").

This paper investigates format biases in Reinforcement Learning from Human Feedback (RLHF), a crucial technique for aligning large language models (LLMs) with human preferences. The authors observe that many widely-used preference models, including human evaluators, GPT-4, and top-ranking models on benchmarks, exhibit strong biases towards specific format patterns such as lists, links, bold text, and emojis.

The study analyzes various preference datasets and benchmarks, including RLHFlow-Preference-700K, LMSYS-Arena-55K, AlpacaEval, and UltraFeedback. They find that these biases are prevalent across different datasets and models. For instance, GPT-4 shows a strong preference for longer sentences, bold formatting, lists, exclamation marks, emojis, hyperlinks, and an affirmative tone.

The paper further demonstrates that even a small amount of biased data (less than 1%) can significantly influence the reward model during training. For example, adding just 0.7% of data with a list bias to the training set increased the model's preference for lists from 51% to 77.5%. This bias then propagates to downstream alignment tasks, such as best-of-n sampling and online iterative DPO, where LLMs can easily exploit these biases to achieve higher rewards without actually improving the content quality. For example, in the best-of-n sampling experiment, when the reward model was trained on a dataset with both bold and list biases, the ratio of bold patterns in generated responses increased from 42.8% to 51.9%, and the ratio of list patterns increased from 57.1% to 64.4% as the number of candidate responses increased.

The authors argue that these findings highlight the importance of disentangling format and content in both the design of alignment algorithms and the evaluation of LLMs. They suggest that simply removing training data with specific patterns is insufficient to address the issue. Instead, future work should focus on developing more robust reward models and alignment algorithms that are less susceptible to format biases.

Conclusion

This newsletter has showcased the latest advancements in RLHF, highlighting its transformative potential across various domains. From enabling non-expert guidance in multi-agent systems to aligning generative models with human preferences, RLHF is paving the way for more reliable, robust, and aligned AI systems. However, as evident in the discussion on format biases, careful consideration of potential pitfalls, such as bias propagation and the need for robust evaluation metrics, is crucial to ensure the responsible and ethical development of this powerful technology. As research in RLHF continues to advance, we can anticipate even more innovative applications and a deeper understanding of how to effectively integrate human feedback into the learning process, shaping the future of AI systems that are not only intelligent but also aligned with human values.