pyspark.
BarrierTaskContext
A TaskContext with extra contextual info and tooling for tasks in a barrier stage. Use BarrierTaskContext.get() to obtain the barrier context for a running barrier task.
TaskContext
BarrierTaskContext.get()
New in version 2.4.0.
Notes
This API is experimental
Methods
allGather([message])
allGather
This function blocks until all tasks in the same stage have reached this routine.
attemptNumber()
attemptNumber
”
barrier()
barrier
Sets a global barrier and waits until all tasks in this stage hit this barrier.
get()
get
Return the currently active BarrierTaskContext.
getLocalProperty(key)
getLocalProperty
Get a local property set upstream in the driver, or None if it is missing.
getTaskInfos()
getTaskInfos
Returns BarrierTaskInfo for all tasks in this barrier stage, ordered by partition ID.
BarrierTaskInfo
partitionId()
partitionId
The ID of the RDD partition that is computed by this task.
resources()
resources
Resources allocated to the task.
stageId()
stageId
The ID of the stage that this task belong to.
taskAttemptId()
taskAttemptId
An ID that is unique to this task attempt (within the same SparkContext, no two task attempts will share the same attempt ID).