This tutorial will explain with examples how to use array_union, array_intersect and array_except array functions in Pyspark.

PySpark: Dataframe Array Functions Part 2

This tutorial will explain with examples how to use array_union, array_intersect and array_except array functions in Pyspark. Other array functions can be viewed by clicking functions in the below list.

• array_union
• array_intersect
• array_except
• array_join
• array_sort
• array_position
• array_contains
• array_remove
• array_distinct
• array_min
• array_max
• array_repeat
• arrays_overlap
• arrays_zip

Sample Data: Dataset used in the below examples can be downloaded from here.

from pyspark.sql import Row

df = spark.createDataFrame([Row(col1=["b", "a", "c",None], col2=["c", "d", "a", "f"])])

df.show()
+---------+------------+
|     col1|        col2|
+---------+------------+
|[b, a, c]|[c, d, a, f]|
+---------+------------+

➠ array_union: This function can be used to return array of combined elements from 2 array columns without duplicates. It is available to import from Pyspark Sql function library.

• Syntax:

  
  array_union(column1, column2)
  

  • → 1st parameter(column1) takes a column name containing array.
  • → 2nd parameter(column2) also takes a column name containing array.
  
  
• Example 1: 2 columns were combined into single column.

  import pyspark.sql.functions as f
  
  df_updated = df.select(f.array_union(df.col1, df.col2).alias("union_"))
  
  df_updated.show()
  +---------------+
  |         union_|
  +---------------+
  |[b, a, c, d, f]|
  +---------------+
  

  
  
• Example 2: Array union with null

  import pyspark.sql.functions as f
  df = spark.createDataFrame([Row(col1=["b", "a", "c",None], col2=["c", "d", "a", "f"])])
  
  df.show()
  +----------+------------+
  |      col1|        col2|
  +----------+------------+
  |[b, a, c,]|[c, d, a, f]|
  +----------+------------+
  
  
  df_updated = df.select(f.array_union(df.col1, df.col2).alias("union_"))
  
  
  df_updated.show()
  +----------------+
  |          union_|
  +----------------+
  |[b, a, c,, d, f]|
  +----------------+
  

➠ array_intersect: This function can be used to return array of common elements from 2 array columns without duplicates. It is available to import from Pyspark Sql function library.

• Syntax:

  
  array_intersect(column1, column2)
  

  • → 1st parameter(column1) takes a column name containing array.
  • → 2nd parameter(column2) also takes a column name containing array.
  
  
• Example 1: 2 columns were combined into single column.

  import pyspark.sql.functions as f
  
  df_updated = df.select(f.array_intersect(df.col1, df.col2).alias("array_intersect_"))
  
  df_updated.show()
  +----------------+
  |array_intersect_|
  +----------------+
  |          [a, c]|
  +----------------+
  

➠ array_except: This function can be used to return array of elements which are present in first column and not in second column without duplicates. It is available to import from Pyspark Sql function library.

• Syntax:

  
  array_except(column1, column2)
  

  • → 1st parameter(column1) takes a column name containing array.
  • → 2nd parameter(column2) also takes a column name containing array.
  
  
• Example 1: Non common element was return from first column

  import pyspark.sql.functions as f
  
  df_updated = df.select(f.array_except(df.col1, df.col2).alias("array_except_"))
  
  df_updated.show()
  +-------------+
  |array_except_|
  +-------------+
  |          [b]|
  +-------------+
  

  
  
• Example 2: array_except with null

  import pyspark.sql.functions as f
  df = spark.createDataFrame([Row(col1=["b", "a", "c",None], col2=["c", "d", "a", "f"])])
  
  df.show()
  +----------+------------+
  |      col1|        col2|
  +----------+------------+
  |[b, a, c,]|[c, d, a, f]|
  +----------+------------+
  
  
  df_updated = df.select(f.array_except(df.col1, df.col2).alias("array_except_"))
  
  
  df_updated.show()
  +-------------+
  |array_except_|
  +-------------+
  |         [b,]|
  +-------------+