Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
(Submitted on 11 Feb 2024 (v1), last revised 25 Apr 2024 (this version, v2))
Abstract: We study Reinforcement Learning from Human Feedback (RLHF) under a general preference oracle. In particular, we do not assume that there exists a reward function and the preference signal is drawn from the Bradley-Terry model as most of the prior works do. We consider a standard mathematical formulation, the reverse-KL regularized minimax game between two LLMs for RLHF under general preference oracle. The learning objective of this formulation is to find a policy so that it is consistently preferred by the KL-regularized preference oracle over any competing LLMs. We show that this framework is strictly more general than the reward-based one, and propose sample-efficient algorithms for both the offline learning from a pre-collected preference dataset and online learning where we can query the preference oracle along the way of training. Empirical studies verify the effectiveness of the proposed framework.
Submission history
From: Chenlu Ye [view email][v1] Sun, 11 Feb 2024 21:44:21 GMT (65kb)
[v2] Thu, 25 Apr 2024 04:05:06 GMT (65kb,D)
Link back to: arXiv, form interface, contact.