InstructGPT
InstructGPT is a language processing artificial intelligence developed by OpenAI that is designed to be more aligned with user intentions and produce less toxic and untruthful output compared to the company's GPT-3 model. InstructGPT is trained using a technique called reinforcement learning from human feedback (RLHF), in which human annotators provide demonstrations of desired model behavior and rank model outputs. These rankings are then used to fine-tune GPT-3 and create the InstructGPT model. The resulting model is able to more accurately follow instructions and produce outputs that are preferred by human labelers, while still maintaining strong performance on academic natural language processing evaluations.
Highlights
- InstructGPT is a more aligned and safer language model
OpenAI's InstructGPT models are trained to better follow user instructions while generating less toxic or harmful output compared to GPT-3. - InstructGPT generates more helpful and appropriate outputs
InstructGPT outputs are preferred by labelers over GPT-3 outputs and are more appropriate with less made-up facts and fewer imitative falsehoods. - InstructGPT reduces harmful and biased outputs
InstructGPT models are fine-tuned with human feedback to mitigate the harms and biases of language models on protected classes. - InstructGPT models are trained using reinforcement learning
InstructGPT models are trained using reinforcement learning from human feedback to fine-tune GPT-3 models and maximize reward using PPO algorithm. - InstructGPT shows small decreases in toxic output
Compared to GPT-3, InstructGPT generates fewer imitative falsehoods and is less toxic according to existing metrics such as TruthfulQA and RealToxicity Prompts - InstructGPT is more aligned with customer tasks
InstructGPT models are trained on a dataset of human-labeled comparisons on API prompts and are preferred over FLAN4 and T0.5 models on customer distribution - InstructGPT reduces the alignment tax
During RL fine-tuning, InstructGPT mixes in a small fraction of data used to train GPT-3 to mitigate performance decreases on academic NLP tasks
Top 16 Alternatives To "InstructGPT"
Submit an entry
Is there an entry, which should be added to this list?