We’re excited to announce DATABRICKS RUNTIME OUTPERFORMS OTHER COMPUTE ENGINES
“>Databricks Runtime 4.2, powered by Apache Spark™. Version 4.2 includes updated Spark internals, new features, and major performance upgrades to Databricks Delta, as well as general quality improvements to the platform. We are moving quickly toward the Databricks Delta general availability (GA) release and we recommend you upgrade to Databricks Runtime 4.2 to take advantage of these improvements.
I’d like to take a moment to highlight some of the work the team has done to continually improve Databricks Delta:
- Streaming Directly to Delta Tables: Streams can now be directly written to a Databricks Delta table registered in the Hive metastore using df.writeStream.table(…).
- Path Consistency for Delta Commands: All Databricks Delta commands and queries now support referring to a table using its path as an identifier (that is, delta.`/path/to/table`). Previously OPTIMIZE and VACUUM required non-standard use of string literals (that is, ‘/path/to/table’).
We’ve also included powerful new features to Trigger.Once in the Databricks Runtime 4.2 release notes.
We included support for the SQL Deny command for table access control enabled clusters. Users can now deny specific permissions in the same way they are granted. A denied permission will supersede a granted one. Detailed technical documentation is available at: SQL DENY.
To read more about the above new features and to see the full list of improvements included in Databricks Runtime 4.2, please refer to the release notes in the following locations: