JNJoubin Najmaieindatafragments.com·Jun 21, 2024 · 2 min readWeek of June 17 2024 - Mindmap RecapJune 17, 2024: Databricks - Issues with Excel Library in Clusters An issue was encountered with the crealytics:spark-excel library in Databricks. This Spark plugin is essential for reading and writing Excel files within Databricks. However, we observ...00
JNJoubin Najmaieindatafragments.com·Jun 9, 2024 · 2 min readWeek of June 3 2024 - Mindmap RecapThe importance of abstraction, reusability, error handling, efficient data manipulation, robust string handling, and performance optimization. Adopting these principles leads to cleaner, more maintainable, and high-performance code that becomes cruci...00
JNJoubin Najmaieindatafragments.com·Jan 23, 2022 · 1 min readSearch All Databricks' Workspace NotebooksAs we accumulate library of sample and reference code through various Databricks notebooks finding the code is not easy. The Purpose behind this notebook is to find the code/text in the Databricks' workspace easier and navigate to it. You can downloa...00
JNJoubin Najmaieindatafragments.com·Jan 16, 2022 · 1 min readEnrich Streaming Data with Batched DataStreaming data can be enriched using the following scenarios: Static References Dynamic Data Sets Another Streaming Data Source This post covers two of the above scenarios: 1 and 2. First setup the streaming reader outputPath = f'{working_dir}/outp...00
JNJoubin Najmaieindatafragments.com·Dec 28, 2021 · 1 min readDatabricks Temp Views and CachingThere are two kinds of temp views: Session based Global The temp views, once created, are not registered in the underlying metastore. The non-global (session) temp views are session based and are purged when the session ends. The global temp views ...00
