Spark job driver OOM

Hey yall I’m upgrading my spark script from 3.5 to 4.1 and all of a sudden the same job is failing due to driver OOM. I’m not running collect or anything just a simple count(). Around 100k task are created and the driver OOMs about 25% through. I’m giving it about 8gb memory. Any idea what’s going on? Did something change in spark 4? submitted by /u/Inevitable_Mango592 [link] [comments]