Language Models are Few-Shot Learners (OpenAI GPT-3)

Language Models are Few-Shot Learners (OpenAI GPT-3)

Machine Learning Street Talk  Tim Scarfe, Yannic Kilcher and Connor Shorten discuss their takeaways from OpenAI’s GPT-3 language model.

OpenAI trained a 175 BILLION parameter autoregressive language model. The paper demonstrates how self-supervised language modelling at this scale can perform many downstream tasks without fine-tuning. 

Paper Links:

Content index:

  • 00:00:00 Intro
  • 00:00:54 ZeRO1+2 (model + Data parallelism) [GPT-3 DOES *NOT* USE THIS] (Connor)
  • 00:03:17 Recent history of NLP (Tim)
  • 00:06:04 Yannic “Light-speed” Kilcher’s brief overview of GPT-3
  • 00:14:25 Reviewing Yannic’s YT comments on his GPT-3 video (Tim)
  • 00:20:26 Main show intro
  • 00:23:03 Is GPT-3 reasoning?
  • 00:28:15 Architecture discussion and autoregressive (GPT*) vs denoising autoencoder (BERT)
  • 00:36:18 Utility of GPT-3 in industry
  • 00:43:03 Can GPT-3 do math? (reasoning/system 1/system 2)
  • 00:51:03 Generalisation
  • 00:56:48 Esoterics of language models
  • 00:58:46 Architectural trade-offs
  • 01:07:37 Memorization machines and intepretability
  • 01:17:16 Nearest neighbour probes / watermarks
  • 01:20:03 YouTube comments on GPT-3 video
  • 01:21:50 GPT-3 news article generation issue
  • 01:27:36 Sampling data for language models / bias / fairness / politics
  • 01:51:12 Outro

GPT-3: Language Models are Few-Shot Learners

GPT-3: Language Models are Few-Shot Learners

How far can you go with ONLY language modeling?

Can a large enough language model perform NLP task out of the box?

OpenAI take on these and other questions by training a transformer that is an order of magnitude larger than anything that has ever been built before and the results are astounding.

Yannic Kilcher explores.

Paper

Time index:

  • 0:00 – Intro & Overview
  • 1:20 – Language Models
  • 2:45 – Language Modeling Datasets
  • 3:20 – Model Size
  • 5:35 – Transformer Models
  • 7:25 – Fine Tuning
  • 10:15 – In-Context Learning
  • 17:15 – Start of Experimental Results
  • 19:10 – Question Answering
  • 23:10 – What I think is happening
  • 28:50 – Translation
  • 31:30 – Winograd Schemes
  • 33:00 – Commonsense Reasoning
  • 37:00 – Reading Comprehension
  • 37:30 – SuperGLUE
  • 40:40 – NLI
  • 41:40 – Arithmetic Expressions
  • 48:30 – Word Unscrambling
  • 50:30 – SAT Analogies
  • 52:10 – News Article Generation
  • 58:10 – Made-up Words
  • 1:01:10 – Training Set Contamination
  • 1:03:10 – Task Exampleshttps://arxiv.org/abs/2005.14165
    https://github.com/openai/gpt-3

Will Robot Writers Change the Internet?

Will Robot Writers Change the Internet?

Computers just got a lot better at mimicking human language. Researchers created computer programs that can write long passages of coherent, original text.

Language models like GPT-2, Grover, and CTRL create text passages that seem written by someone fluent in the language, but not in the truth. That AI field, Natural Language Processing (NLP), didn’t exactly set out to create a fake news machine. Rather, it’s the byproduct of a line of research into massive pretrained language models: Machine learning programs that store vast statistical maps of how we use our language. So far, the technology’s creative uses seem to outnumber its malicious ones. But it’s not difficult to imagine how these text-fakes could cause harm, especially as these models become widely shared and deployable by anyone with basic know-how.

Read more here: https://www.vox.com/recode/2020/3/4/21163743/ai-language-generation-fake-text-gpt2 

Should you be an AI Generalist or Specialist?

Should you be an AI Generalist or Specialist?

Machine Learning with Phil ponders the question: “is it better to specialize or generalize in artificial intelligence and deep learning?”

The answer depends on your career aspirations. Do you want to be a deep learning research professor?

Do you want to go to work for Google, Facebook, or other global mega corporations?

Or do you want to be your own unicorn start up founder?

Each has their own specialization requirements that Phil breaks down in this video.

Sentiment Analysis With Tensorflow 2

Sentiment Analysis With Tensorflow 2

Machine Learning with Phil show you how to do sentiment analysis with TensorFlow 2 in this natural language processing (NLP) tutorial.

This natural language processing model is relatively straight forward, as it’s just an encoder coupled to some bidirectional layers and a couple dense layers to handle the classification. We’ll compare two different models, one with a single LSTM layer and the other with two LSTM layers and some dropout.