Data Science Wire

Spark plus Neon db, how much JDBC parallelism is too much ?

Reddit r/apachespark1w4 min read

Have u benchmarked writing to Neon over JDBC conn with diff numPartitions/connection counts? I was exploring how Neon behaves when Spark starts opening a lot of parallel connection, does it improve throughput, or do we hit connection /IO bottlenecks quickly? Curious what people have found to work well in practice. submitted by /u/sqlink2 [link] [comments]

Read the full story at Reddit r/apachespark

More in Data Engineering