How far can you go with ONLY language modeling?

Can a large enough language model perform NLP task out of the box?

OpenAI take on these and other questions by training a transformer that is an order of magnitude larger than anything that has ever been built before and the results are astounding.

Yannic Kilcher explores.

Paper

Time index:

  • 0:00 – Intro & Overview
  • 1:20 – Language Models
  • 2:45 – Language Modeling Datasets
  • 3:20 – Model Size
  • 5:35 – Transformer Models
  • 7:25 – Fine Tuning
  • 10:15 – In-Context Learning
  • 17:15 – Start of Experimental Results
  • 19:10 – Question Answering
  • 23:10 – What I think is happening
  • 28:50 – Translation
  • 31:30 – Winograd Schemes
  • 33:00 – Commonsense Reasoning
  • 37:00 – Reading Comprehension
  • 37:30 – SuperGLUE
  • 40:40 – NLI
  • 41:40 – Arithmetic Expressions
  • 48:30 – Word Unscrambling
  • 50:30 – SAT Analogies
  • 52:10 – News Article Generation
  • 58:10 – Made-up Words
  • 1:01:10 – Training Set Contamination
  • 1:03:10 – Task Exampleshttps://arxiv.org/abs/2005.14165
    https://github.com/openai/gpt-3

Over the past decade, prices for solar panels and wind farms have reached all-time lows.

However, the price for lithium ion batteries, the leading energy storage technology, has remained high.

So researchers are exploring other alternatives, including flow batteries, thermal batteries, and gravity-based systems.

Commercially viable quantum computing could be here sooner than you think, thanks to a new innovation that shrinks quantum tech down onto a chip: a cryochip.

Seeker explains:

It seems like quantum computers will likely be a big part of our computing future—but getting them to do anything super useful has been famously difficult. Lots of new technologies are aiming to get commercially viable quantum computing here just a little bit faster, including one innovation that shrinks quantum technology down onto a chip.

The Infographics Show provides a look at the industries hit hardest by the COVID shutdown.

With nearly the whole world in quarantine, more business are being forced to close their doors, and some may never re-open. In today’s video we’re going to look at the industries affected most by the global pandemic. Places like hotels, music venues, and movie studios have all closed down. What ripple effects will these massive shut downs cause to the economy? Watch today’s informative video to find out which work places have been hit the hardest by this insane pandemic.

RNZ interviews Jacob Glanville, one of the stars of Netflix documentary Pandemic. He runs Distributed Bio which has been working to find an antibody therapy.

Yesterday he tweeted we should get ready for a positive announcement this week. He joins Lisa Owen via Skype from San Francisco.Scientists around the world have been racing to develop treatments, cures and a vaccine for COVID-19 – and are getting closer by the day.

Microsoft’s Project Silica aims to show that glass is the future of long-term data storage.

To prove its usefulness outside the lab, Microsoft partnered with Warner Bros. to write the 1978 Superman film into glass with lasers.

To see the whole process and the Superman glass, CNET visited Microsoft’s Research Lab in Cambridge, England and Warner Bros. Studios in Burbank, California.

In October 2019, Google announced its 53-qubit quantum computer named Sycamore had achieved ‘quantum supremacy.’

That’s when quantum computers can complete tasks exponentially more quickly than their classical counterparts.

In this case, Google said its quantum machine completed a task in 200 seconds that would have taken the world’s most powerful computer 10,000 years to complete. IBM, another major player in quantum computing, took issue with the findings.

Either way, it was a big milestone in quantum computing, and it’s leading to a lot of hype in the field. Here’s how quantum computing works, and how it could change everything from Wall Street to Big Pharma and beyond.