pyspark.RDD.withResources#

RDD.withResources(profile)[source]#

Specify a pyspark.resource.ResourceProfile to use when calculating this RDD. This is only supported on certain cluster managers and currently requires dynamic allocation to be enabled. It will result in new executors with the resources specified being acquired to calculate the RDD.

New in version 3.1.0.

Parameters
profilepyspark.resource.ResourceProfile

a resource profile

Returns
RDD

the same RDD with user specified profile

See also

RDD.getResourceProfile()

Notes

This API is experimental