In this talk from the Databricks YouTube Channel is about date-time processing in Spark 3.0, its API and implementations made since Spark 2.4.
In particular,it covers the following topics:
- Definition and internal representation of dates/timestamps in Spark SQL. Comparisons of Spark 3.0 date-time API with previous versions and other DBMS.
- Date/timestamp functions of Spark SQL. Nuances of behavior and details of implementation. Use cases and corner cases of date-time API.
- Migration from the hybrid calendar (Julian and Gregorian calendars) to Proleptic Gregorian calendar in Spark 3.0.
- Parsing of date/timestamp strings, saving and loading date/time data via Spark’s datasources.
- Support of Java 8 time API in Spark 3.0.