LinkedIn has just open sourced a tool it created to convert Apache Spark-based big data into a format that can be readily consumed by TensorFlow. The tool, Avro2TF “bridges the gap and presents an elegant solution for ML engineers, freeing them up to focus on different deep learning algorithms,” the team stated.
The team noted that like LinkedIn, many other companies have vast amounts of ML data in a similar sparse vector format, and Tensor format is still relatively new to many of them. The believe that Avro2TF can help address these challenges.
It’s interesting to see LinkedIn, which itself is owned by Microsoft, open source their internal AI/ML tooling, joining the likes of Uber, Facebook, etc. What a time to be alive! 🙂