• Spark
Well.
I’ve spent several hours, days… trying to pull some data from my tables. But my query joining 2 tables to fill the Null values doesn’t match with pySpark.
Null and Nan and blanks are different in Spark… Once I figured that out, my quest to replace some of the values.
It’s fun to apply regex with Spark.
With a lot of tables, it’s better to select the column I need and include it in my select statement.
The ‘withColumn(), ‘when()’ and ‘otherwise()’ allow me to get null values from a dictionnary.