Half Ideas – Startups and Entrepreneurship takes a closer look at GPT-3 and what it means for AI.

GPT 3 can write poetry, translate text, chat convincingly, and answer abstract questions. It’s being used to code, design and much more. I’ll give you a demo of some of the latest in this technology and some of how it works.

GPT 3 has been developed for a number of years. One of the early papers published was on Generative Pre-Training. The idea behind generative pre-training (GPT) is that while most AI’s are trained on labeled data, there’s a ton of data that isn’t labeled. If you can evaluate the words and use them to train and tune the AI it can start to create predictions of future text on the unlabeled data. You repeat the process until predictions start to converge.   

On AiRR interviewed me recently and asked what my “one word” to program an AGI (Artificial General Intelligence) would be.

The word is “#Exist”a great talk with Frank Lavigne a Data and AI Architect from Microsoft Corp. We spoke about the harmony between the nature and Artificial General Intelligence, It was a very fun chat !

Machine Learning Street Talk  Tim Scarfe, Yannic Kilcher and Connor Shorten discuss their takeaways from OpenAI’s GPT-3 language model.

OpenAI trained a 175 BILLION parameter autoregressive language model. The paper demonstrates how self-supervised language modelling at this scale can perform many downstream tasks without fine-tuning. 

Paper Links:

Content index:

  • 00:00:00 Intro
  • 00:00:54 ZeRO1+2 (model + Data parallelism) [GPT-3 DOES *NOT* USE THIS] (Connor)
  • 00:03:17 Recent history of NLP (Tim)
  • 00:06:04 Yannic “Light-speed” Kilcher’s brief overview of GPT-3
  • 00:14:25 Reviewing Yannic’s YT comments on his GPT-3 video (Tim)
  • 00:20:26 Main show intro
  • 00:23:03 Is GPT-3 reasoning?
  • 00:28:15 Architecture discussion and autoregressive (GPT*) vs denoising autoencoder (BERT)
  • 00:36:18 Utility of GPT-3 in industry
  • 00:43:03 Can GPT-3 do math? (reasoning/system 1/system 2)
  • 00:51:03 Generalisation
  • 00:56:48 Esoterics of language models
  • 00:58:46 Architectural trade-offs
  • 01:07:37 Memorization machines and intepretability
  • 01:17:16 Nearest neighbour probes / watermarks
  • 01:20:03 YouTube comments on GPT-3 video
  • 01:21:50 GPT-3 news article generation issue
  • 01:27:36 Sampling data for language models / bias / fairness / politics
  • 01:51:12 Outro

On AiRR is a new podcast that talks about a the rise of AGI, artificial general intelligence, in the context of one word.

The discussions are a delight to listen to.

The word is “Responsibility” Artificial General Intelligence, One of the very important principles is going to be responsibility interms of A.I. Ethics and also when it comes to people working in A.I. ! The AGI is bound to be responsible and that is what Roldan and Rupesh speak about in this episode with Michelle Zhou CEO and Co Founder of Juji.io The matter of Ethics in A.I. is very serious!