Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (Explained)

#gpt3 #knowledge #symbolic Symbolic knowledge models are usually trained on human-generated corpora that are cumbersome and expensive to create. Such corpora consist of structured triples of symbolic knowledge. This paper takes a different approach and attempts to generate such a corpus by prompting GPT-3. Results show that clever prompting, combined with targeted small critic models trained on human ratings can outperform both human-generated data, as well as the teacher model (GPT-3) itself. The results of this paper give a general recipe for automatically building corpora for various NLP tasks by extracting samples from large language models. OUTLINE: 0:00 - Intro & Overview 2:30 - Sponsor: Weights & Biases 4:15 - Commonsense Knowledge Graphs 7:50 - ATOMIC dataset 10:00 - Generating the corpus from a model 13:00 - Prompting GPT-3 15:30 - Generating Events 18:40 - Generating Inferences 23:00 - Evaluating the created dataset 26:45 - Introducing the critic 31:25 - Using the critic to filter the data 36:30 -