In this talk from the Databricks YouTube Channel is about date-time processing in Spark 3.0, its API and implementations made since Spark 2.4.

In particular,it covers the following topics:

  1. Definition and internal representation of dates/timestamps in Spark SQL. Comparisons of Spark 3.0 date-time API with previous versions and other DBMS.
  2. Date/timestamp functions of Spark SQL.  Nuances of behavior and details of implementation. Use cases and corner cases of date-time API.
  3. Migration from the hybrid calendar (Julian and Gregorian calendars) to Proleptic Gregorian calendar in Spark 3.0.
  4. Parsing of date/timestamp strings, saving and loading date/time data via Spark’s datasources.
  5. Support of Java 8 time API in Spark 3.0.

tt ads