pyspark.sql.streaming.StreamingQuery.explain#

StreamingQuery.explain(extended=False)[source]#

Prints the (logical and physical) plans to the console for debugging purpose.

New in version 2.1.0.

Changed in version 3.5.0: Supports Spark Connect.

Parameters

extendedbool, optional: default False. If False, prints only the physical plan.

Examples

>>> sdf = spark.readStream.format("rate").load()
>>> sdf.printSchema()
root
  |-- timestamp: timestamp (nullable = true)
  |-- value: long (nullable = true)

>>> sq = sdf.writeStream.format('memory').queryName('query_explain').start()
>>> sq.processAllAvailable() # Wait a bit to generate the runtime plans.

Explain the runtime plans

>>> sq.explain()
== Physical Plan ==
...
>>> sq.explain(True)
== Parsed Logical Plan ==
...
== Analyzed Logical Plan ==
...
== Optimized Logical Plan ==
...
== Physical Plan ==
...
>>> sq.stop()