This tutorial will explain how you can preview, display or print 'n' rows on the console from the Spark dataframe.

PySpark: Dataframe Preview (Part 1)

This tutorial will explain how you can preview, display or print 'n' rows on the console from the Spark dataframe. Below listed dataframe functions will be explained with examples, click on function name in the below list and it will take you to the respective section of the function:

• Show
• Head
• Tail
• First
• Take
• Collect with Limit
  
  Sample Data: Dataset (contains 50 rows) used in the below examples can be downloaded from here (orders).

  
  df=spark.read.parquet("file:///path_to_file/orders.parquet")
  
  df.show(truncate=False)
  +--------+---------------------+-----------------+---------------+
  |order_id|order_date           |order_customer_id|order_status   |
  +--------+---------------------+-----------------+---------------+
  |68817   |2014-03-27 00:00:00.0|6704             |COMPLETE       |
  |68818   |2014-03-31 00:00:00.0|12393            |PROCESSING     |
  .
  .
  .
  |68835   |2014-05-02 00:00:00.0|764              |COMPLETE       |
  |68836   |2014-05-03 00:00:00.0|8009             |PENDING_PAYMENT|
  +--------+---------------------+-----------------+---------------+
  only showing top 20 rows
  

➠ Show: show() function can be used to display / print first n rows from dataframe on the console in a tabular format. Show function can take up to 3 parameters and all 3 parameters are optional.

dataframe.show(n=20, truncate=True, vertical=False)

• 1st parameter 'n' is used to specify the number of rows that will be shown. Default value for this optional parameter is 20.
• 2nd parameter 'truncate' can be used to specify whether to truncate column data or not in the tabular format. This parameter can take either True or False as possible value. By default, truncate parameter is True and strings longer than 20 characters will be truncated.
• 3rd parameter 'vertical' can be used to show rows vertically i.e. one line per column value per row. This parameter can take either True or False as possible value. By default, vertical parameter is False and all columns from same row will be on same line.

• Example 1: show() function without any parameter will show at most 20 rows and truncates column value if it is more than 20 chars.

  
  df.show()
  
  +--------+--------------------+-----------------+---------------+
  |order_id|          order_date|order_customer_id|   order_status|
  +--------+--------------------+-----------------+---------------+
  |   68817|2014-03-27 00:00:...|             6704|       COMPLETE|
  |   68818|2014-03-31 00:00:...|            12393|     PROCESSING|
  .
  .
  .
  |   68835|2014-05-02 00:00:...|              764|       COMPLETE|
  |   68836|2014-05-03 00:00:...|             8009|PENDING_PAYMENT|
  +--------+--------------------+-----------------+---------------+
  only showing top 20 rows
  

  
  
• Example 2: show() function with 'n' numbered parameter will show that many number of rows from the dataframe.

  
  df.show(2)
  
  +--------+--------------------+-----------------+------------+
  |order_id|          order_date|order_customer_id|order_status|
  +--------+--------------------+-----------------+------------+
  |   68817|2014-03-27 00:00:...|             6704|    COMPLETE|
  |   68818|2014-03-31 00:00:...|            12393|  PROCESSING|
  +--------+--------------------+-----------------+------------+
  only showing top 2 rows
  

  
  
• Example 3: select() function can be used along with show() function to preview dataframe partially for some columns.

  
  df.select("order_id","order_status").show(2)
  
  +--------+---------------+
  |order_id|   order_status|
  +--------+---------------+
  |   68817|       COMPLETE|
  |   68818|     PROCESSING|
  +--------+---------------+
  only showing top 2 rows
  

  
  
• Example 4: show() can be used to specify truncate parameter along with 'n' numbered parameter(1st parameter).

  
  df.show(4,False)
  
  +--------+---------------------+-----------------+------------+
  |order_id|order_date           |order_customer_id|order_status|
  +--------+---------------------+-----------------+------------+
  |68817   |2014-03-27 00:00:00.0|6704             |COMPLETE    |
  |68818   |2014-03-31 00:00:00.0|12393            |PROCESSING  |
  |68819   |2014-04-03 00:00:00.0|1212             |COMPLETE    |
  |68820   |2014-04-04 00:00:00.0|6358             |COMPLETE    |
  +--------+---------------------+-----------------+------------+
  only showing top 4 rows
  

  
  
• Example 5: truncate key/value should be passed to show() if truncate parameter is required without 'n' numbered parameter(1st parameter).

  
  df.show(truncate=False)
  
  +--------+---------------------+-----------------+---------------+
  |order_id|order_date           |order_customer_id|order_status   |
  +--------+---------------------+-----------------+---------------+
  |68817   |2014-03-27 00:00:00.0|6704             |COMPLETE       |
  |68818   |2014-03-31 00:00:00.0|12393            |PROCESSING     |
  .
  .
  .
  |68835   |2014-05-02 00:00:00.0|764              |COMPLETE       |
  |68836   |2014-05-03 00:00:00.0|8009             |PENDING_PAYMENT|
  +--------+---------------------+-----------------+---------------+
  only showing top 20 rows
  

  
  
• Example 6: Using limit function along with show() function.

  
  df.limit(1).show()
  
  +--------+--------------------+-----------------+------------+
  |order_id|          order_date|order_customer_id|order_status|
  +--------+--------------------+-----------------+------------+
  |   68817|2014-03-27 00:00:...|             6704|    COMPLETE|
  +--------+--------------------+-----------------+------------+
  

  
  
• Example 7: Showing all the rows present in the dataframe.

  
  df.show(df.count())
  
  +--------+--------------------+-----------------+---------------+
  |order_id|          order_date|order_customer_id|   order_status|
  +--------+--------------------+-----------------+---------------+
  |   68817|2014-03-27 00:00:...|             6704|       COMPLETE|
  |   68818|2014-03-31 00:00:...|            12393|     PROCESSING|
  .
  .
  .
  |   68799|2014-02-14 00:00:...|            11190|       COMPLETE|
  |   68800|2014-02-17 00:00:...|            10037|     PROCESSING|
  +--------+--------------------+-----------------+---------------+
  

  
  
• Example 8: Vertical key/value should be passed to show() if it is the only parameter used.

  
  df.show(2,vertical=True)
  
  -RECORD 0---------------------------------
   order_id          | 68817                
   order_date        | 2014-03-27 00:00:... 
   order_customer_id | 6704                 
   order_status      | COMPLETE             
  -RECORD 1---------------------------------
   order_id          | 68818                
   order_date        | 2014-03-31 00:00:... 
   order_customer_id | 12393                
   order_status      | PROCESSING           
  only showing top 2 rows
  

  
  
• Example 9: All 3 parameters can be used together in the show() function.

  
  df.show(2, truncate=False, vertical=True)
  
  -RECORD 0----------------------------------
   order_id          | 68817                 
   order_date        | 2014-03-27 00:00:00.0 
   order_customer_id | 6704                  
   order_status      | COMPLETE              
  -RECORD 1----------------------------------
   order_id          | 68818                 
   order_date        | 2014-03-31 00:00:00.0 
   order_customer_id | 12393                 
   order_status      | PROCESSING            
  only showing top 2 rows