This tutorial will explain with examples how to use array_sort and array_join array functions in Pyspark.

PySpark: Dataframe Array Functions Part 1

This tutorial will explain with examples how to use array_sort and array_join array functions in Pyspark. Other array functions can be viewed by clicking functions in the below list.

• array_join
• array_sort
• array_union
• array_intersect
• array_except
• array_position
• array_contains
• array_remove
• array_distinct
• array_min
• array_max
• array_repeat
• arrays_overlap
• arrays_zip

Sample Data: Dataset used in the below examples can be downloaded from here.

df = spark.createDataFrame([(["d","a", "b", "c"],), (["f","a", None],)], ['data'])

df.show()
+------------+
|        data|
+------------+
|[d, a, b, c]|
|     [f, a,]|
+------------+

➠ array_sort: This function can be used to sort elements of array column in ascending order, Null/None elements will be placed at the end of the returned array. It is available to import from Pyspark Sql function library.

• Syntax:

  
  array_sort(column)
  

  • → 1st parameter(column) takes a column name on which this function need to be applied.
  
  
• Example 1: Elements of array column was sorted in ascending order.

  import pyspark.sql.functions as f
  
  df_updated = df.select(df.data, f.array_sort(df.data).alias("sorted"))
  
  df_updated.show()
  +------------+------------+
  |        data|      sorted|
  +------------+------------+
  |[d, a, b, c]|[a, b, c, d]|
  |     [f, a,]|     [a, f,]|
  +------------+------------+
  

➠ array_join: This function can be used to concatenate elements of array column using a string delimiter. It is available to import from Pyspark Sql function library.

• Syntax:

  
  array_join(column, delimiter, null_replacement=None)
  

  • → 1st parameter(column) takes a column name on which this function need to be applied.
  • → 2nd parameter(delimiter) takes a string value to specify whether to cache data in the memory or not.
  • → 3rd parameter(null_replacement) is optional and takes a string value to replace any null / None values in the array with the string passed in this parameter.
  
  
• Example 1: Column's array values are concatenated using comma (,).

  import pyspark.sql.functions as f
  
  df_updated = df.select(df.data, f.array_join(df.data, ",").alias("joined"))
  
  df_updated.show()
  +------------+-------+
  |        data| joined|
  +------------+-------+
  |[d, a, b, c]|d,a,b,c|
  |     [f, a,]|    f,a|
  +------------+-------+
  

  
  
• Example 2: Column's array values are concatenated using pipe (|).

  import pyspark.sql.functions as f
  
  df_updated = df.select(df.data, f.array_join(df.data, "|").alias("joined"))
  
  df_updated.show()
  +------------+-------+
  |        data| joined|
  +------------+-------+
  |[d, a, b, c]|d|a|b|c|
  |     [f, a,]|    f|a|
  +------------+-------+
  

  
  
• Example 3: Column's array values are concatenated using underscore (_) and nulls are replaced with asterisk(*) using 3rd parameter.

  import pyspark.sql.functions as f
  
  df_updated = df.select(df.data, f.array_join(df.data, "_","*").alias("joined"))
  
  df_updated.show()
  +------------+-------+
  |        data| joined|
  +------------+-------+
  |[d, a, b, c]|d_a_b_c|
  |     [f, a,]|  f_a_*|
  +------------+-------+
  

  
  
• Example 4: Array join function is used along with the array sort function.

  import pyspark.sql.functions as f
  
  df_updated = df.select(df.data, f.array_join(f.array_sort(df.data), "_").alias("ascending_and_joined"))
  
  df_updated.show()
  +------------+--------------------+
  |        data|ascending_and_joined|
  +------------+--------------------+
  |[d, a, b, c]|             a_b_c_d|
  |     [f, a,]|                 a_f|
  +------------+--------------------+