PySpark Dataframe Sampling

This tutorial will explain how to use different sample functions available in Pyspark to extract subset of dataframe from the main dataframe. There are multiple dataframe functions for data sampling, click on function name in the below list and it will take you to the respective section of the page.

sample : Used for random sampling
sampleBy : Used for column based sampling

Users can visit this page if they only want to preview the data.

Sample Data:

here


empdf = spark.read.parquet("file:///path_to_files/employee.parquet")

empdf.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000245| PRADEEP|5000.00|      null|    100|
|1000258|   BLAKE|2850.00|   1000245|    300|
|1000262|   CLARK|2450.00|   1000245|    100|
|1000276|   JONES|2975.00|   1000245|    200|
|1000288|   SCOTT|3000.00|   1000276|    200|
|1000292|    FORD|3000.00|   1000276|    200|
|1000294|   SMITH| 800.00|   1000292|    200|
|1000299|   ALLEN|1600.00|   1000258|    300|
|1000310|    WARD|1250.00|   1000258|    300|
|1000312|  MARTIN|1250.00|   1000258|    300|
|1000315|  TURNER|1500.00|   1000258|    300|
|1000326|   ADAMS|1100.00|   1000288|    200|
|1000336|   JAMES| 950.00|   1000258|    300|
|1000346|  MILLER|1300.00|   1000262|    100|
|1000347|   DAVID|1400.00|   1000245|    500|
+-------+--------+-------+----------+-------+

➠ Sample: sample function can be used for random sampling of dataframe.

Syntax:


sample(withReplacement=None, fraction=None, seed=None)

→ The first parameter (withReplacement) takes the boolean value and with the value as True, it can have duplicates in the sample dataframe.
→ The second parameter (fraction) can be used to specify the fraction of rows to be generated for the sample dataframe. Fraction value may range from 0.0 to 1.0 decimal.
→ The third parameter (seed) takes the numeric value as input and with the same seed number, it will return the same output for each run otherwise it will be a random output.

Note:

Example 1: If only one parameter is passed with a value between(0.0 and 1.0), Spark will take that as a fraction parameter.


empdf_sample = empdf.sample(0.2)

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000262|   CLARK|2450.00|   1000245|    100|
+-------+--------+-------+----------+-------+

Example 2: As mentioned above with the replacement parameter as True, it can have duplicates in sample dataframe.


empdf_sample = empdf.sample(True, 0.3)

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000288|   SCOTT|3000.00|   1000276|    200|
|1000292|    FORD|3000.00|   1000276|    200|
|1000312|  MARTIN|1250.00|   1000258|    300|
|1000312|  MARTIN|1250.00|   1000258|    300|
+-------+--------+-------+----------+-------+

Example 3: With the same seed value, spark will return the same dataframe subset.


empdf_sample = empdf.sample(True,0.4,3)

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000292|    FORD|3000.00|   1000276|    200|
|1000294|   SMITH| 800.00|   1000292|    200|
|1000315|  TURNER|1500.00|   1000258|    300|
|1000346|  MILLER|1300.00|   1000262|    100|
+-------+--------+-------+----------+-------+



empdf_sample = empdf.sample(0.4,2)

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000258|   BLAKE|2850.00|   1000245|    300|
|1000292|    FORD|3000.00|   1000276|    200|
|1000312|  MARTIN|1250.00|   1000258|    300|
|1000326|   ADAMS|1100.00|   1000288|    200|
|1000336|   JAMES| 950.00|   1000258|    300|
|1000346|  MILLER|1300.00|   1000262|    100|
+-------+--------+-------+----------+-------+



empdf_sample = empdf.sample(0.4,2)

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000258|   BLAKE|2850.00|   1000245|    300|
|1000292|    FORD|3000.00|   1000276|    200|
|1000312|  MARTIN|1250.00|   1000258|    300|
|1000326|   ADAMS|1100.00|   1000288|    200|
|1000336|   JAMES| 950.00|   1000258|    300|
|1000346|  MILLER|1300.00|   1000262|    100|
+-------+--------+-------+----------+-------+

Example 4: All 3 parameters are used in this example.


empdf_sample = empdf.sample(True,0.4,3)

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000292|    FORD|3000.00|   1000276|    200|
|1000294|   SMITH| 800.00|   1000292|    200|
|1000315|  TURNER|1500.00|   1000258|    300|
|1000346|  MILLER|1300.00|   1000262|    100|
+-------+--------+-------+----------+-------+

➠ SampleBy: sampleBy function can be used for column value(s) based sampling of dataframe.

Syntax:


sampleBy(column, fractions, seed=None)

→ The first parameter (column) takes the column (name) that can be used to generate sample dataframe.
→ The second parameter (fractions) takes dictionary of column value and fraction to generate sample dataframe accordingly. Fraction values may range from 0.0 to 1.0 decimal.
→ The third parameter (seed) takes the numeric value as input and with the same seed number, it will return the same output for each run otherwise it will be a random output.

Note:

Example 1: In the below example, 200 and 300 in the dictionary are the department number(dept_no) and 0.1 & 0.2 are the fraction requested for the respective department. Also, since seed parameter is not used, each run may give different sample dataframe but will have rows only with the column value passed in the dictionary.


empdf_sample = empdf.sampleBy("dept_no", {200:0.1,300:0.2})

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000310|    WARD|1250.00|   1000258|    300|
|1000312|  MARTIN|1250.00|   1000258|    300|
+-------+--------+-------+----------+-------+



empdf_sample = empdf.sampleBy("dept_no", {200:0.1,300:0.2})

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000258|   BLAKE|2850.00|   1000245|    300|
|1000294|   SMITH| 800.00|   1000292|    200|
|1000299|   ALLEN|1600.00|   1000258|    300|
|1000310|    WARD|1250.00|   1000258|    300|
|1000312|  MARTIN|1250.00|   1000258|    300|
|1000336|   JAMES| 950.00|   1000258|    300|
+-------+--------+-------+----------+-------+



empdf_sample = empdf.sampleBy("dept_no", {200:0.1,300:0.2})

empdf_sample.show()
+-------+--------+------+----------+-------+
| emp_no|emp_name|salary|manager_id|dept_no|
+-------+--------+------+----------+-------+
|1000294|   SMITH|800.00|   1000292|    200|
+-------+--------+------+----------+-------+

Example 2: With the same seed value, spark will return the same dataframe subset.


empdf_sample = empdf.sampleBy("dept_no", {200:0.1,300:0.2},1)

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000276|   JONES|2975.00|   1000245|    200|
|1000299|   ALLEN|1600.00|   1000258|    300|
+-------+--------+-------+----------+-------+



empdf_sample = empdf.sampleBy("dept_no", {200:0.1,300:0.2},1)

empdf_sample.show()
+-------+--------+-------+----------+-------+
| emp_no|emp_name| salary|manager_id|dept_no|
+-------+--------+-------+----------+-------+
|1000276|   JONES|2975.00|   1000245|    200|
|1000299|   ALLEN|1600.00|   1000258|    300|
+-------+--------+-------+----------+-------+

This tutorial will explain how to use different sample functions available in Pyspark to extract subset of dataframe from the main dataframe.

dbmstutorials.com

PySpark Dataframe Sampling