This tutorial will explain with examples how to use array_distinct, array_min, array_max and array_repeat array functions in Pyspark.

PySpark: Dataframe Array Functions Part 4

This tutorial will explain with examples how to use array_distinct, array_min, array_max and array_repeat array functions in Pyspark. Other array functions can be viewed by clicking functions in the below list.

• array_distinct
• array_min
• array_max
• array_repeat
• array_join
• array_sort
• array_union
• array_intersect
• array_except
• array_position
• array_contains
• array_remove
• arrays_overlap
• arrays_zip

Sample Data: Dataset used in the below examples can be downloaded from here.

df = spark.createDataFrame([(["d","a", "b","a", "c"],), (["f","d","a", None],)], ['data'])

df.show()
+---------------+
|           data|
+---------------+
|[d, a, b, a, c]|
|     [f, d, a,]|
+---------------+

➠ array_distinct: This function can be used to remove duplicate values from array column. It is available to import from Pyspark Sql function library.

• Syntax:

  
  array_distinct(column)
  

  • → 1st parameter(column) takes a column name containing array.
  
  
  
• Example 1: First occurence was kept and all other duplicate occurence were removed.

  import pyspark.sql.functions as f
  
  df_updated = df.select(df.data, f.array_distinct(df.data).alias("distinct_data"))
  
  df_updated.show()
  +---------------+-------------+
  |           data|distinct_data|
  +---------------+-------------+
  |[d, a, b, a, c]| [d, a, b, c]|
  |     [f, d, a,]|   [f, d, a,]|
  +---------------+-------------+
  

➠ array_min: This function can be used to returns the minimum value of the array. It is available to import from Pyspark Sql function library.

• Syntax:

  
  array_min(column)
  

  • → 1st parameter(column) takes a column name containing array.
  
  
  
• Example 1: Minimum value was returned

  import pyspark.sql.functions as f
  
  df_updated = df.select(df.data, f.array_min(df.data).alias("minimum_data"))
  
  df_updated.show()
  +---------------+------------+
  |           data|minimum_data|
  +---------------+------------+
  |[d, a, b, a, c]|           a|
  |     [f, d, a,]|           a|
  +---------------+------------+
  

➠ array_max: This function can be used to returns the maximum value of the array. It is available to import from Pyspark Sql function library.

• Syntax:

  
  array_max(column)
  

  • → 1st parameter(column) takes a column name containing array.
  
  
  
• Example 1: Maximum value was returned.

  import pyspark.sql.functions as f
  
  df_updated = df.select(df.data, f.array_max(df.data).alias("maximum_data"))
  
  df_updated.show()
  +---------------+------------+
  |           data|maximum_data|
  +---------------+------------+
  |[d, a, b, a, c]|           d|
  |     [f, d, a,]|           f|
  +---------------+------------+
  

➠ array_repeat: This function can be used to return array containing a column repeated specified number of times. It is available to import from Pyspark Sql function library.

• Syntax:

  
  array_repeat(column, count)
  

  • → 1st parameter(column) takes a column name containing array.
  
  
  
• Example 1: Column was repeated 3 times.

  import pyspark.sql.functions as f
  
  df_updated = df.select(df.data, f.array_repeat(df.data, 3).alias("repeat_data"))
  
  df_updated.show(truncate=False)
  +---------------+---------------------------------------------------+
  |data           |repeat_data                                        |
  +---------------+---------------------------------------------------+
  |[d, a, b, a, c]|[[d, a, b, a, c], [d, a, b, a, c], [d, a, b, a, c]]|
  |[f, d, a,]     |[[f, d, a,], [f, d, a,], [f, d, a,]]               |
  +---------------+---------------------------------------------------+