Why might increasing the number of Spark shuffle partitions help with out-of-memory errors during a large aggregation, but also potentially hurt performance if set too high?

A More partitions mean smaller per-task data, reducing memory pressure, but too many small partitions add scheduling and task-overhead costs that can slow the job down B Shuffle partition count has absolutely no effect on memory usage or performance C More partitions always strictly improve performance with no downside whatsoever D Increasing shuffle partitions always reduces the total amount of data processed