This tutorial will explain how to write Spark dataframe into various types of comma separated value(CSV) files or other delimited files.

PySpark: Dataframe To File(Part 1)

This tutorial will explain how to write Spark dataframe into various types of comma separated value(CSV) files or other delimited files.


Write CSV file(without header): By default, sparks creates a comma separated values file without header if write operation is invoked on dataframe.
CSV file(with header): Spark provides a way to write header columns as 1st row to a file(s) using either option() or options() functions. Options function is used in the below example.

Write Delimited file: Although CSV files are also delimited files, these examples are separately mentioned here to write delimited files with customized separator i.e delimiter other than comma(,).
Write CSV to HDFS: Spark can also write data to HDFS system. As such there is no syntax difference in writing to Local / server or HDFS, only difference will be the path.

Write File with Specific Name: Writing directly as file is not supported in Spark but there are workarounds.