Was ChatGPT a good idea?

In this week’s ML & AI Safety Update, we hear Paul Christiano’s take on one of OpenAI’s main alignment strategies, dive into the second round winners of the inverse scaling prize and share the many fascinating projects from our mechanistic interpretability hackathon. And stay tuned until the end for some unique opportunities in AI safety!

Watch this week's MLAISU on YouTube or listen to it on Spotify.

Reinforcement learning from human feedback

Reinforcement learning from human feedback (RLHF) is one of the most applied techniques from alignment research. Its history started in 2015 when Paul Christiano introduced the concept in a blog post.

The idea is that we train models not just to imitate humans, but also to act in ways that humans would evaluate as preferable. This basic idea has resulted in years of research at OpenAI and is now one of the main principles behind ChatGPT.

Two days ago, Christiano published a piece evaluating the impact of RLHF on the speed-up of AGI versus progress on aligning said AGI. He thinks the project has been net positive and that replacements that work as well in practice (e.g. imitation learning) would have been used for AI capabilities unless RLHF was developed.

Additionally, Christiano counters arguments from the AI safety community, mentioning that RLHF is:

• Safer than alternatives and showcases the risks of ML systems without the necessary scale-up in AI technology.
• Is not inherently unique capabilities-wise and is able to produce realistic examples of deeper problems with large models.

Inverse scaling prize

The inverse scaling prize has found its second round winners in a challenge to find tasks where larger language models such as GPT-3 do worse than GPT-2. These are generally hard to find and they are very important to identify to figure out which abilities will fail in larger models more generally.

The seven winners of the second round have all used quite novel method to get there:

1. Modus Tollens is a task to identify if a statement is true or false. An example might be “If John has a pet, then John has a dog. John does not have a dog. Therefore, John doesn’t have a pet. Is the conclusion correct?”. Surprisingly, larger models become worse at answering that yes, this conclusion is correct.
2. Memo Trap shows that larger models have a tendency to end famous quotes with the quote text despite explicit instructions to end the quote differently. This is also true for biased quotes from “racist Jim Crow laws and homophobic Bible verses”.
3. Prompt Injection works to input a malicious prompt injection that overrides previous instructions. Interestingly, medium-sized models are most prone to these “textual overrides” than larger models, and it shows a performance over model size that is U-shaped!

I recommend checking out the other four winners in their report on the round 2 projects.

Alignment Jam 4

​The Fourth Alignment Jam ended this Sunday, with 15 amazing projects submitted! It was on the topic of “mechanistic interpretability”, where we try to reverse engineer how neural networks (NN) process input. Since NNs learn algorithms from the training data, we can actually try to find specific algorithms for specific tasks within the network.

You can watch the ending ceremony with presentations by three of the four winners (starts here) but here is a short summarization of the winning projects:

1. In “We Discovered An Neuron”, Miller and Neo used the TransformerLens library to find an MLP neuron in GPT-2 large that predicts the token “ an” and dives deep into how it works and when it activates using activation patching, ablation, and other methods.
2. ​Mathwin and Corlouer used the Automatic Circuit Discovery tool from Arthur Conmy to identify circuits for gendered pronouns. It is a wonderful example of using the tools we have available to automatically identify circuits and understand them in-depth.
3. ​Michelle Wai Man Lo created a new way to identify feature neurons automatically by identifying which tokens neurons activate for and automatically generating descriptions for what they do! In this way, we can get descriptions of most neurons in a smaller network within a few hours.
4. ​The Mentaleap team found that the embedding space for prompt tuning tasks is convex! What this means is that we can add multiple tokens together as a replacement for another token for specific tasks.

It was tough deciding the winners together with Neel Nanda and you can see many more in the results section of the hackathon page. We recommend you check them out! There’s methods from biology, compiled Transformers, interactive apps, and latent knowledge identification methods.

Opportunities

With the help of AGISF and AI Safety Support, we’re sharing some amazing opportunities this week!

• The deadline to join a biology and social systems fellowship for AI safety is coming up in 10 days (PIBBBS)!
• ​The Effective Altruism Global conferences are coming up with a big one in London in May. You can get free tickets to the event and get to know other AI safety interested people and experts.
• ​Join the ML safety introduction course from the Center for AI Safety!
• ​The Alignment Awards competitions are a great way to engage with AI safety while potentially winning from the $50,000 prize pool! There’s a challenge on making sure AI systems generalize well and one on making sure we can update AI systems after they are deployed.

Thank you for following along for this week’s ML & AI Safety Update and we’ll see you next week!