Here’s an interesting article that I found via LinkedIn.

Mind Machine learning experts struggle to deal with “overfitting” in neural networks. Evolution solved it with dreams, says new theory. The world of sport is filled with superstition. Michael Jordan famously wore University of North Carolina shorts under his Chicago Bulls kit; Serena Williams wears the same socks throughout […]

Visual scenes are often comprised of sets of independent objects. Yet, current vision models make no assumptions about the nature of the pictures they look at.

Yannic Kilcher explore a paper on object-centric learning.

By imposing an objectness prior, this paper a module that is able to recognize permutation-invariant sets of objects from pixels in both supervised and unsupervised settings. It does so by introducing a slot attention module that combines an attention mechanism with dynamic routing.

Content index:

  • 0:00 – Intro & Overview
  • 1:40 – Problem Formulation
  • 4:30 – Slot Attention Architecture
  • 13:30 – Slot Attention Algorithm
  • 21:30 – Iterative Routing Visualization
  • 29:15 – Experiments
  • 36:20 – Inference Time Flexibility
  • 38:35 – Broader Impact Statement
  • 42:05 – Conclusion & Comments

Yannic Kilcher retraces his first reading of Facebook AI’s DETR paper and explain my process of understanding it.

OUTLINE:

  • 0:00 – Introduction
  • 1:25 – Title
  • 4:10 – Authors
  • 5:55 – Affiliation
  • 7:40 – Abstract
  • 13:50 – Pictures
  • 20:30 – Introduction
  • 22:00 – Related Work
  • 24:00 – Model
  • 30:00 – Experiments
  • 41:50 – Conclusions & Abstract
  • 42:40 – Final Remarks

Original Video about DETR: https://youtu.be/T35ba_VXkMY

Text-to-speech engines are usually multi-stage pipelines that transform the signal into many intermediate representations and require supervision at each step.

When trying to train TTS end-to-end, the alignment problem arises: Which text corresponds to which piece of sound?

This paper uses an alignment module to tackle this problem and produces astonishingly good sound.

Paper: https://arxiv.org/abs/2006.03575
Website: https://deepmind.com/research/publications/End-to-End-Adversarial-Text-to-Speech

Content index:

  • 0:00 – Intro & Overview
  • 1:55 – Problems with Text-to-Speech
  • 3:55 – Adversarial Training
  • 5:20 – End-to-End Training
  • 7:20 – Discriminator Architecture
  • 10:40 – Generator Architecture
  • 12:20 – The Alignment Problem
  • 14:40 – Aligner Architecture
  • 24:00 – Spectrogram Prediction Loss
  • 32:30 – Dynamic Time Warping
  • 38:30 – Conclusion

Yannic Kilcher explores a recent innovation at Facebook

Code migration between languages is an expensive and laborious task. To translate from one language to the other, one needs to be an expert at both. Current automatic tools often produce illegible and complicated code. This paper applies unsupervised neural machine translation to source code of Python, C++, and Java and is able to translate between them, without ever being trained in a supervised fashion.

Paper: https://arxiv.org/abs/2006.03511

Content index:

  • 0:00 – Intro & Overview
  • 1:15 – The Transcompiling Problem
  • 5:55 – Neural Machine Translation
  • 8:45 – Unsupervised NMT
  • 12:55 – Shared Embeddings via Token Overlap
  • 20:45 – MLM Objective
  • 25:30 – Denoising Objective
  • 30:10 – Back-Translation Objective
  • 33:00 – Evaluation Dataset
  • 37:25 – Results
  • 41:45 – Tokenization
  • 42:40 – Shared Embeddings
  • 43:30 – Human-Aware Translation
  • 47:25 – Failure Cases
  • 48:05 – Conclusion

How far can you go with ONLY language modeling?

Can a large enough language model perform NLP task out of the box?

OpenAI take on these and other questions by training a transformer that is an order of magnitude larger than anything that has ever been built before and the results are astounding.

Yannic Kilcher explores.

Paper

Time index:

  • 0:00 – Intro & Overview
  • 1:20 – Language Models
  • 2:45 – Language Modeling Datasets
  • 3:20 – Model Size
  • 5:35 – Transformer Models
  • 7:25 – Fine Tuning
  • 10:15 – In-Context Learning
  • 17:15 – Start of Experimental Results
  • 19:10 – Question Answering
  • 23:10 – What I think is happening
  • 28:50 – Translation
  • 31:30 – Winograd Schemes
  • 33:00 – Commonsense Reasoning
  • 37:00 – Reading Comprehension
  • 37:30 – SuperGLUE
  • 40:40 – NLI
  • 41:40 – Arithmetic Expressions
  • 48:30 – Word Unscrambling
  • 50:30 – SAT Analogies
  • 52:10 – News Article Generation
  • 58:10 – Made-up Words
  • 1:01:10 – Training Set Contamination
  • 1:03:10 – Task Exampleshttps://arxiv.org/abs/2005.14165
    https://github.com/openai/gpt-3

MIT Introduction to Deep Learning 6.S191: Lecture 6 with Ava Soleimany.

Subscribe to stay up to date with new deep learning lectures at MIT, or follow us @MITDeepLearning on Twitter and Instagram to stay fully-connected!!

Lecture Outline

  • 0:00 – Introduction
  • 0:58 – Course logistics
  • 3:59 – Upcoming guest lectures
  • 5:35 – Deep learning and expressivity of NNs
  • 10:02 – Generalization of deep models
  • 14:14 – Adversarial attacks
  • 17:00 – Limitations summary
  • 18:18 – Structure in deep learning
  • 22:53 – Uncertainty & bayesian deep learning
  • 28:09 – Deep evidential regression
  • 33:08 – AutoML
  • 36:43 – Conclusion

This tutorial on TensorFlow.org implements a simplified Quantum Convolutional Neural Network (QCNN), a proposed quantum analogue to a classical convolutional neural network that is also translationally invariant.

Wow.

This tutorial implements a simplified Quantum Convolutional Neural Network (QCNN), a proposed quantum analogue to a classical convolutional neural network that is also translationally invariant . This example demonstrates how to detect certain properties of a quantum data source, such as a quantum sensor or a complex simulation from […]