Which method is used to split the data across folders when saving a dataframe?

• A. splitBy
• B. distributeBy
• C. partitionBy
• D. bucketBy

Answer: (C)

Partitioning the output is about organizing saved data into folders based on one or more column values. When you use the partitioning feature on the writer, Spark writes the data into subdirectories named after the partition columns, such as path/category=value/date=value. This creates a folder structure like category=Books/date=2024-01-01/ with the actual data files inside. It helps with query performance because engines can prune away entire folders when filters on the partition columns are present, avoiding reading irrelevant data.

For example, df.write.partitionBy("country", "year").parquet("/data/output") will split the saved files into folders for each country and year combination. This is different from bucketing (which distributes data into a fixed number of buckets within each partition) or other write-time distribution tricks, which don’t automatically create that hierarchical folder layout. The partitionBy method is specifically what controls the split across folders.