In [21]:
!jupyter nbconvert \
/root/volumes/source.ml/jupyterhub.ml/notebooks/spark/pyspark_pi.ipynb \
--to script \
--output-dir=/root/volumes/source.ml/jupyterhub.ml/scripts/spark/
[NbConvertApp] Converting notebook /root/volumes/source.ml/jupyterhub.ml/notebooks/spark/pyspark_pi.ipynb to script
[NbConvertApp] Writing 501 bytes to /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py
In [19]:
cat /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py
# coding: utf-8
# In[7]:
import sys
from random import random
from operator import add
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("PythonPi").getOrCreate()
partitions = 100
n = 100000 * partitions
def f(_):
x = random() * 2 - 1
y = random() * 2 - 1
return 1 if x ** 2 + y ** 2 <= 1 else 0
count = spark.sparkContext.parallelize(range(1, n + 1), partitions).map(f).reduce(add)
print("Pi is roughly %f" % (4.0 * count / n))
spark.stop()
# In[ ]:
In [20]:
!spark-submit --master spark://spark-master-2-1-0:7077 \
/root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/03/08 22:35:17 INFO SparkContext: Running Spark version 2.0.1
17/03/08 22:35:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/03/08 22:35:18 INFO SecurityManager: Changing view acls to: root
17/03/08 22:35:18 INFO SecurityManager: Changing modify acls to: root
17/03/08 22:35:18 INFO SecurityManager: Changing view acls groups to:
17/03/08 22:35:18 INFO SecurityManager: Changing modify acls groups to:
17/03/08 22:35:18 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
17/03/08 22:35:18 INFO Utils: Successfully started service 'sparkDriver' on port 44222.
17/03/08 22:35:18 INFO SparkEnv: Registering MapOutputTracker
17/03/08 22:35:18 INFO SparkEnv: Registering BlockManagerMaster
17/03/08 22:35:18 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-c1cfea49-8443-4058-8193-203ec2752771
17/03/08 22:35:18 INFO MemoryStore: MemoryStore started with capacity 373.2 MB
17/03/08 22:35:18 INFO SparkEnv: Registering OutputCommitCoordinator
17/03/08 22:35:18 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
17/03/08 22:35:18 INFO Utils: Successfully started service 'SparkUI' on port 4041.
17/03/08 22:35:18 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://100.96.9.57:4041
17/03/08 22:35:19 INFO SparkContext: Added file file:/root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py at spark://100.96.9.57:44222/files/pyspark_pi.py with timestamp 1489012519024
17/03/08 22:35:19 INFO Utils: Copying /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py to /tmp/spark-2f7edac3-6d2a-438f-ac46-8302914167b2/userFiles-cea17686-125f-49cb-b468-9ce41e440c86/pyspark_pi.py
17/03/08 22:35:19 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark-master-2-0-1:7077...
17/03/08 22:35:19 INFO TransportClientFactory: Successfully created connection to spark-master-2-0-1/100.67.230.192:7077 after 27 ms (0 ms spent in bootstraps)
17/03/08 22:35:19 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20170308223519-0028
17/03/08 22:35:19 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170308223519-0028/0 on worker-20170306194937-100.96.9.14-41412 (100.96.9.14:41412) with 1 cores
17/03/08 22:35:19 INFO StandaloneSchedulerBackend: Granted executor ID app-20170308223519-0028/0 on hostPort 100.96.9.14:41412 with 1 cores, 1024.0 MB RAM
17/03/08 22:35:19 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170308223519-0028/0 is now RUNNING
17/03/08 22:35:19 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41194.
17/03/08 22:35:19 INFO NettyBlockTransferService: Server created on 100.96.9.57:41194
17/03/08 22:35:19 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 100.96.9.57, 41194)
17/03/08 22:35:19 INFO BlockManagerMasterEndpoint: Registering block manager 100.96.9.57:41194 with 373.2 MB RAM, BlockManagerId(driver, 100.96.9.57, 41194)
17/03/08 22:35:19 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 100.96.9.57, 41194)
17/03/08 22:35:19 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/03/08 22:35:19 INFO SharedState: Warehouse path is '/root/volumes/source.ml/jupyterhub.ml/notebooks/spark/spark-warehouse'.
17/03/08 22:35:20 INFO SparkContext: Starting job: reduce at /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py:22
17/03/08 22:35:20 INFO DAGScheduler: Got job 0 (reduce at /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py:22) with 100 output partitions
17/03/08 22:35:20 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py:22)
17/03/08 22:35:20 INFO DAGScheduler: Parents of final stage: List()
17/03/08 22:35:20 INFO DAGScheduler: Missing parents: List()
17/03/08 22:35:20 INFO DAGScheduler: Submitting ResultStage 0 (PythonRDD[1] at reduce at /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py:22), which has no missing parents
17/03/08 22:35:21 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.2 KB, free 373.2 MB)
17/03/08 22:35:21 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.5 KB, free 373.2 MB)
17/03/08 22:35:21 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 100.96.9.57:41194 (size: 3.5 KB, free: 373.2 MB)
17/03/08 22:35:21 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1012
17/03/08 22:35:21 INFO DAGScheduler: Submitting 100 missing tasks from ResultStage 0 (PythonRDD[1] at reduce at /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py:22)
17/03/08 22:35:21 INFO TaskSchedulerImpl: Adding task set 0.0 with 100 tasks
17/03/08 22:35:22 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (100.96.9.14:41200) with ID 0
17/03/08 22:35:22 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 100.96.9.14, partition 0, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:22 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 0 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:22 INFO BlockManagerMasterEndpoint: Registering block manager 100.96.9.14:33032 with 373.2 MB RAM, BlockManagerId(0, 100.96.9.14, 33032)
17/03/08 22:35:23 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 100.96.9.14:33032 (size: 3.5 KB, free: 373.2 MB)
17/03/08 22:35:24 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 100.96.9.14, partition 1, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:24 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 1 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:24 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1792 ms on 100.96.9.14 (1/100)
17/03/08 22:35:24 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, 100.96.9.14, partition 2, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:24 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 2 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:24 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 194 ms on 100.96.9.14 (2/100)
17/03/08 22:35:24 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, 100.96.9.14, partition 3, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:24 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 3 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:24 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 223 ms on 100.96.9.14 (3/100)
17/03/08 22:35:24 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, 100.96.9.14, partition 4, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:24 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 4 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:24 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 276 ms on 100.96.9.14 (4/100)
17/03/08 22:35:25 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, 100.96.9.14, partition 5, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:25 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 5 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:25 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 158 ms on 100.96.9.14 (5/100)
17/03/08 22:35:25 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, 100.96.9.14, partition 6, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:25 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 166 ms on 100.96.9.14 (6/100)
17/03/08 22:35:25 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 6 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:25 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, 100.96.9.14, partition 7, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:25 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 7 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:25 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 173 ms on 100.96.9.14 (7/100)
17/03/08 22:35:25 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, 100.96.9.14, partition 8, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:25 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 8 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:25 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 203 ms on 100.96.9.14 (8/100)
17/03/08 22:35:25 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, 100.96.9.14, partition 9, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:25 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 9 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:25 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 184 ms on 100.96.9.14 (9/100)
17/03/08 22:35:25 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, 100.96.9.14, partition 10, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:25 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 10 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:25 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 191 ms on 100.96.9.14 (10/100)
17/03/08 22:35:26 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, 100.96.9.14, partition 11, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:26 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 11 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:26 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 199 ms on 100.96.9.14 (11/100)
17/03/08 22:35:26 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, 100.96.9.14, partition 12, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:26 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 12 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:26 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 150 ms on 100.96.9.14 (12/100)
17/03/08 22:35:26 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, 100.96.9.14, partition 13, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:26 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 13 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:26 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 184 ms on 100.96.9.14 (13/100)
17/03/08 22:35:26 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, 100.96.9.14, partition 14, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:26 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 14 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:26 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 215 ms on 100.96.9.14 (14/100)
17/03/08 22:35:26 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, 100.96.9.14, partition 15, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:26 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 15 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:26 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 237 ms on 100.96.9.14 (15/100)
17/03/08 22:35:27 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, 100.96.9.14, partition 16, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:27 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 16 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:27 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 164 ms on 100.96.9.14 (16/100)
17/03/08 22:35:27 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, 100.96.9.14, partition 17, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:27 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 17 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:27 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 173 ms on 100.96.9.14 (17/100)
17/03/08 22:35:27 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, 100.96.9.14, partition 18, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:27 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 18 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:27 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 148 ms on 100.96.9.14 (18/100)
17/03/08 22:35:27 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, 100.96.9.14, partition 19, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:27 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 19 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:27 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 217 ms on 100.96.9.14 (19/100)
17/03/08 22:35:27 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, 100.96.9.14, partition 20, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:27 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 20 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:27 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 203 ms on 100.96.9.14 (20/100)
17/03/08 22:35:27 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, 100.96.9.14, partition 21, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:27 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 21 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:27 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 151 ms on 100.96.9.14 (21/100)
17/03/08 22:35:28 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, 100.96.9.14, partition 22, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:28 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 22 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:28 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 178 ms on 100.96.9.14 (22/100)
17/03/08 22:35:28 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, 100.96.9.14, partition 23, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:28 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 23 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:28 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 215 ms on 100.96.9.14 (23/100)
17/03/08 22:35:28 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, 100.96.9.14, partition 24, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:28 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 24 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:28 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 182 ms on 100.96.9.14 (24/100)
17/03/08 22:35:28 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, 100.96.9.14, partition 25, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:28 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 25 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:28 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 179 ms on 100.96.9.14 (25/100)
17/03/08 22:35:28 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, 100.96.9.14, partition 26, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:28 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 26 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:28 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 167 ms on 100.96.9.14 (26/100)
17/03/08 22:35:29 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 27, 100.96.9.14, partition 27, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:29 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 27 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:29 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 160 ms on 100.96.9.14 (27/100)
17/03/08 22:35:29 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 28, 100.96.9.14, partition 28, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:29 INFO TaskSetManager: Finished task 27.0 in stage 0.0 (TID 27) in 187 ms on 100.96.9.14 (28/100)
17/03/08 22:35:29 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 28 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:29 INFO TaskSetManager: Starting task 29.0 in stage 0.0 (TID 29, 100.96.9.14, partition 29, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:29 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 29 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:29 INFO TaskSetManager: Finished task 28.0 in stage 0.0 (TID 28) in 197 ms on 100.96.9.14 (29/100)
17/03/08 22:35:29 INFO TaskSetManager: Starting task 30.0 in stage 0.0 (TID 30, 100.96.9.14, partition 30, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:29 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 30 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:29 INFO TaskSetManager: Finished task 29.0 in stage 0.0 (TID 29) in 274 ms on 100.96.9.14 (30/100)
17/03/08 22:35:29 INFO TaskSetManager: Starting task 31.0 in stage 0.0 (TID 31, 100.96.9.14, partition 31, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:29 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 31 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:29 INFO TaskSetManager: Finished task 30.0 in stage 0.0 (TID 30) in 210 ms on 100.96.9.14 (31/100)
17/03/08 22:35:30 INFO TaskSetManager: Starting task 32.0 in stage 0.0 (TID 32, 100.96.9.14, partition 32, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:30 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 32 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:30 INFO TaskSetManager: Finished task 31.0 in stage 0.0 (TID 31) in 192 ms on 100.96.9.14 (32/100)
17/03/08 22:35:30 INFO TaskSetManager: Starting task 33.0 in stage 0.0 (TID 33, 100.96.9.14, partition 33, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:30 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 33 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:30 INFO TaskSetManager: Finished task 32.0 in stage 0.0 (TID 32) in 177 ms on 100.96.9.14 (33/100)
17/03/08 22:35:30 INFO TaskSetManager: Starting task 34.0 in stage 0.0 (TID 34, 100.96.9.14, partition 34, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:30 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 34 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:30 INFO TaskSetManager: Finished task 33.0 in stage 0.0 (TID 33) in 186 ms on 100.96.9.14 (34/100)
17/03/08 22:35:30 INFO TaskSetManager: Starting task 35.0 in stage 0.0 (TID 35, 100.96.9.14, partition 35, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:30 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 35 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:30 INFO TaskSetManager: Finished task 34.0 in stage 0.0 (TID 34) in 247 ms on 100.96.9.14 (35/100)
17/03/08 22:35:30 INFO TaskSetManager: Starting task 36.0 in stage 0.0 (TID 36, 100.96.9.14, partition 36, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:30 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 36 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:30 INFO TaskSetManager: Finished task 35.0 in stage 0.0 (TID 35) in 242 ms on 100.96.9.14 (36/100)
17/03/08 22:35:31 INFO TaskSetManager: Starting task 37.0 in stage 0.0 (TID 37, 100.96.9.14, partition 37, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 37 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:31 INFO TaskSetManager: Finished task 36.0 in stage 0.0 (TID 36) in 152 ms on 100.96.9.14 (37/100)
17/03/08 22:35:31 INFO TaskSetManager: Starting task 38.0 in stage 0.0 (TID 38, 100.96.9.14, partition 38, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 38 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:31 INFO TaskSetManager: Finished task 37.0 in stage 0.0 (TID 37) in 158 ms on 100.96.9.14 (38/100)
17/03/08 22:35:31 INFO TaskSetManager: Starting task 39.0 in stage 0.0 (TID 39, 100.96.9.14, partition 39, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 39 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:31 INFO TaskSetManager: Finished task 38.0 in stage 0.0 (TID 38) in 254 ms on 100.96.9.14 (39/100)
17/03/08 22:35:31 INFO TaskSetManager: Starting task 40.0 in stage 0.0 (TID 40, 100.96.9.14, partition 40, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 40 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:31 INFO TaskSetManager: Finished task 39.0 in stage 0.0 (TID 39) in 195 ms on 100.96.9.14 (40/100)
17/03/08 22:35:31 INFO TaskSetManager: Starting task 41.0 in stage 0.0 (TID 41, 100.96.9.14, partition 41, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:31 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 41 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:31 INFO TaskSetManager: Finished task 40.0 in stage 0.0 (TID 40) in 168 ms on 100.96.9.14 (41/100)
17/03/08 22:35:32 INFO TaskSetManager: Starting task 42.0 in stage 0.0 (TID 42, 100.96.9.14, partition 42, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 42 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:32 INFO TaskSetManager: Finished task 41.0 in stage 0.0 (TID 41) in 206 ms on 100.96.9.14 (42/100)
17/03/08 22:35:32 INFO TaskSetManager: Starting task 43.0 in stage 0.0 (TID 43, 100.96.9.14, partition 43, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 43 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:32 INFO TaskSetManager: Finished task 42.0 in stage 0.0 (TID 42) in 147 ms on 100.96.9.14 (43/100)
17/03/08 22:35:32 INFO TaskSetManager: Starting task 44.0 in stage 0.0 (TID 44, 100.96.9.14, partition 44, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 44 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:32 INFO TaskSetManager: Finished task 43.0 in stage 0.0 (TID 43) in 162 ms on 100.96.9.14 (44/100)
17/03/08 22:35:32 INFO TaskSetManager: Starting task 45.0 in stage 0.0 (TID 45, 100.96.9.14, partition 45, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 45 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:32 INFO TaskSetManager: Finished task 44.0 in stage 0.0 (TID 44) in 239 ms on 100.96.9.14 (45/100)
17/03/08 22:35:32 INFO TaskSetManager: Starting task 46.0 in stage 0.0 (TID 46, 100.96.9.14, partition 46, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 46 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:32 INFO TaskSetManager: Finished task 45.0 in stage 0.0 (TID 45) in 174 ms on 100.96.9.14 (46/100)
17/03/08 22:35:32 INFO TaskSetManager: Starting task 47.0 in stage 0.0 (TID 47, 100.96.9.14, partition 47, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:32 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 47 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:32 INFO TaskSetManager: Finished task 46.0 in stage 0.0 (TID 46) in 195 ms on 100.96.9.14 (47/100)
17/03/08 22:35:33 INFO TaskSetManager: Starting task 48.0 in stage 0.0 (TID 48, 100.96.9.14, partition 48, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:33 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 48 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:33 INFO TaskSetManager: Finished task 47.0 in stage 0.0 (TID 47) in 195 ms on 100.96.9.14 (48/100)
17/03/08 22:35:33 INFO TaskSetManager: Starting task 49.0 in stage 0.0 (TID 49, 100.96.9.14, partition 49, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:33 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 49 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:33 INFO TaskSetManager: Finished task 48.0 in stage 0.0 (TID 48) in 172 ms on 100.96.9.14 (49/100)
17/03/08 22:35:33 INFO TaskSetManager: Starting task 50.0 in stage 0.0 (TID 50, 100.96.9.14, partition 50, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:33 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 50 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:33 INFO TaskSetManager: Finished task 49.0 in stage 0.0 (TID 49) in 163 ms on 100.96.9.14 (50/100)
17/03/08 22:35:33 INFO TaskSetManager: Starting task 51.0 in stage 0.0 (TID 51, 100.96.9.14, partition 51, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:33 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 51 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:33 INFO TaskSetManager: Finished task 50.0 in stage 0.0 (TID 50) in 232 ms on 100.96.9.14 (51/100)
17/03/08 22:35:34 INFO TaskSetManager: Starting task 52.0 in stage 0.0 (TID 52, 100.96.9.14, partition 52, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:34 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 52 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:34 INFO TaskSetManager: Finished task 51.0 in stage 0.0 (TID 51) in 269 ms on 100.96.9.14 (52/100)
17/03/08 22:35:34 INFO TaskSetManager: Starting task 53.0 in stage 0.0 (TID 53, 100.96.9.14, partition 53, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:34 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 53 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:34 INFO TaskSetManager: Finished task 52.0 in stage 0.0 (TID 52) in 182 ms on 100.96.9.14 (53/100)
17/03/08 22:35:34 INFO TaskSetManager: Starting task 54.0 in stage 0.0 (TID 54, 100.96.9.14, partition 54, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:34 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 54 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:34 INFO TaskSetManager: Finished task 53.0 in stage 0.0 (TID 53) in 153 ms on 100.96.9.14 (54/100)
17/03/08 22:35:34 INFO TaskSetManager: Starting task 55.0 in stage 0.0 (TID 55, 100.96.9.14, partition 55, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:34 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 55 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:34 INFO TaskSetManager: Finished task 54.0 in stage 0.0 (TID 54) in 142 ms on 100.96.9.14 (55/100)
17/03/08 22:35:34 INFO TaskSetManager: Starting task 56.0 in stage 0.0 (TID 56, 100.96.9.14, partition 56, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:34 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 56 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:34 INFO TaskSetManager: Finished task 55.0 in stage 0.0 (TID 55) in 263 ms on 100.96.9.14 (56/100)
17/03/08 22:35:34 INFO TaskSetManager: Starting task 57.0 in stage 0.0 (TID 57, 100.96.9.14, partition 57, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:34 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 57 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:34 INFO TaskSetManager: Finished task 56.0 in stage 0.0 (TID 56) in 252 ms on 100.96.9.14 (57/100)
17/03/08 22:35:35 INFO TaskSetManager: Starting task 58.0 in stage 0.0 (TID 58, 100.96.9.14, partition 58, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 58 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:35 INFO TaskSetManager: Finished task 57.0 in stage 0.0 (TID 57) in 152 ms on 100.96.9.14 (58/100)
17/03/08 22:35:35 INFO TaskSetManager: Starting task 59.0 in stage 0.0 (TID 59, 100.96.9.14, partition 59, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:35 INFO TaskSetManager: Finished task 58.0 in stage 0.0 (TID 58) in 257 ms on 100.96.9.14 (59/100)
17/03/08 22:35:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 59 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:35 INFO TaskSetManager: Starting task 60.0 in stage 0.0 (TID 60, 100.96.9.14, partition 60, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 60 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:35 INFO TaskSetManager: Finished task 59.0 in stage 0.0 (TID 59) in 234 ms on 100.96.9.14 (60/100)
17/03/08 22:35:35 INFO TaskSetManager: Starting task 61.0 in stage 0.0 (TID 61, 100.96.9.14, partition 61, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 61 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:35 INFO TaskSetManager: Finished task 60.0 in stage 0.0 (TID 60) in 182 ms on 100.96.9.14 (61/100)
17/03/08 22:35:35 INFO TaskSetManager: Starting task 62.0 in stage 0.0 (TID 62, 100.96.9.14, partition 62, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:35 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 62 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:35 INFO TaskSetManager: Finished task 61.0 in stage 0.0 (TID 61) in 148 ms on 100.96.9.14 (62/100)
17/03/08 22:35:36 INFO TaskSetManager: Starting task 63.0 in stage 0.0 (TID 63, 100.96.9.14, partition 63, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:36 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 63 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:36 INFO TaskSetManager: Finished task 62.0 in stage 0.0 (TID 62) in 235 ms on 100.96.9.14 (63/100)
17/03/08 22:35:36 INFO TaskSetManager: Starting task 64.0 in stage 0.0 (TID 64, 100.96.9.14, partition 64, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:36 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 64 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:36 INFO TaskSetManager: Finished task 63.0 in stage 0.0 (TID 63) in 141 ms on 100.96.9.14 (64/100)
17/03/08 22:35:36 INFO TaskSetManager: Starting task 65.0 in stage 0.0 (TID 65, 100.96.9.14, partition 65, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:36 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 65 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:36 INFO TaskSetManager: Finished task 64.0 in stage 0.0 (TID 64) in 150 ms on 100.96.9.14 (65/100)
17/03/08 22:35:36 INFO TaskSetManager: Starting task 66.0 in stage 0.0 (TID 66, 100.96.9.14, partition 66, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:36 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 66 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:36 INFO TaskSetManager: Finished task 65.0 in stage 0.0 (TID 65) in 199 ms on 100.96.9.14 (66/100)
17/03/08 22:35:36 INFO TaskSetManager: Starting task 67.0 in stage 0.0 (TID 67, 100.96.9.14, partition 67, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:36 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 67 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:36 INFO TaskSetManager: Finished task 66.0 in stage 0.0 (TID 66) in 265 ms on 100.96.9.14 (67/100)
17/03/08 22:35:37 INFO TaskSetManager: Starting task 68.0 in stage 0.0 (TID 68, 100.96.9.14, partition 68, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:37 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 68 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:37 INFO TaskSetManager: Finished task 67.0 in stage 0.0 (TID 67) in 148 ms on 100.96.9.14 (68/100)
17/03/08 22:35:37 INFO TaskSetManager: Starting task 69.0 in stage 0.0 (TID 69, 100.96.9.14, partition 69, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:37 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 69 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:37 INFO TaskSetManager: Finished task 68.0 in stage 0.0 (TID 68) in 194 ms on 100.96.9.14 (69/100)
17/03/08 22:35:37 INFO TaskSetManager: Starting task 70.0 in stage 0.0 (TID 70, 100.96.9.14, partition 70, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:37 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 70 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:37 INFO TaskSetManager: Finished task 69.0 in stage 0.0 (TID 69) in 160 ms on 100.96.9.14 (70/100)
17/03/08 22:35:37 INFO TaskSetManager: Starting task 71.0 in stage 0.0 (TID 71, 100.96.9.14, partition 71, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:37 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 71 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:37 INFO TaskSetManager: Finished task 70.0 in stage 0.0 (TID 70) in 185 ms on 100.96.9.14 (71/100)
17/03/08 22:35:37 INFO TaskSetManager: Starting task 72.0 in stage 0.0 (TID 72, 100.96.9.14, partition 72, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:37 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 72 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:37 INFO TaskSetManager: Finished task 71.0 in stage 0.0 (TID 71) in 234 ms on 100.96.9.14 (72/100)
17/03/08 22:35:37 INFO TaskSetManager: Starting task 73.0 in stage 0.0 (TID 73, 100.96.9.14, partition 73, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:37 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 73 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:37 INFO TaskSetManager: Finished task 72.0 in stage 0.0 (TID 72) in 140 ms on 100.96.9.14 (73/100)
17/03/08 22:35:38 INFO TaskSetManager: Starting task 74.0 in stage 0.0 (TID 74, 100.96.9.14, partition 74, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:38 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 74 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:38 INFO TaskSetManager: Finished task 73.0 in stage 0.0 (TID 73) in 159 ms on 100.96.9.14 (74/100)
17/03/08 22:35:38 INFO TaskSetManager: Starting task 75.0 in stage 0.0 (TID 75, 100.96.9.14, partition 75, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:38 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 75 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:38 INFO TaskSetManager: Finished task 74.0 in stage 0.0 (TID 74) in 154 ms on 100.96.9.14 (75/100)
17/03/08 22:35:38 INFO TaskSetManager: Starting task 76.0 in stage 0.0 (TID 76, 100.96.9.14, partition 76, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:38 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 76 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:38 INFO TaskSetManager: Finished task 75.0 in stage 0.0 (TID 75) in 159 ms on 100.96.9.14 (76/100)
17/03/08 22:35:38 INFO TaskSetManager: Starting task 77.0 in stage 0.0 (TID 77, 100.96.9.14, partition 77, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:38 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 77 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:38 INFO TaskSetManager: Finished task 76.0 in stage 0.0 (TID 76) in 206 ms on 100.96.9.14 (77/100)
17/03/08 22:35:38 INFO TaskSetManager: Starting task 78.0 in stage 0.0 (TID 78, 100.96.9.14, partition 78, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:38 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 78 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:38 INFO TaskSetManager: Finished task 77.0 in stage 0.0 (TID 77) in 254 ms on 100.96.9.14 (78/100)
17/03/08 22:35:39 INFO TaskSetManager: Starting task 79.0 in stage 0.0 (TID 79, 100.96.9.14, partition 79, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:39 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 79 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:39 INFO TaskSetManager: Finished task 78.0 in stage 0.0 (TID 78) in 178 ms on 100.96.9.14 (79/100)
17/03/08 22:35:39 INFO TaskSetManager: Starting task 80.0 in stage 0.0 (TID 80, 100.96.9.14, partition 80, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:39 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 80 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:39 INFO TaskSetManager: Finished task 79.0 in stage 0.0 (TID 79) in 179 ms on 100.96.9.14 (80/100)
17/03/08 22:35:39 INFO TaskSetManager: Starting task 81.0 in stage 0.0 (TID 81, 100.96.9.14, partition 81, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:39 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 81 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:39 INFO TaskSetManager: Finished task 80.0 in stage 0.0 (TID 80) in 141 ms on 100.96.9.14 (81/100)
17/03/08 22:35:39 INFO TaskSetManager: Starting task 82.0 in stage 0.0 (TID 82, 100.96.9.14, partition 82, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:39 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 82 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:39 INFO TaskSetManager: Finished task 81.0 in stage 0.0 (TID 81) in 168 ms on 100.96.9.14 (82/100)
17/03/08 22:35:39 INFO TaskSetManager: Starting task 83.0 in stage 0.0 (TID 83, 100.96.9.14, partition 83, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:39 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 83 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:39 INFO TaskSetManager: Finished task 82.0 in stage 0.0 (TID 82) in 186 ms on 100.96.9.14 (83/100)
17/03/08 22:35:40 INFO TaskSetManager: Starting task 84.0 in stage 0.0 (TID 84, 100.96.9.14, partition 84, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:40 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 84 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:40 INFO TaskSetManager: Finished task 83.0 in stage 0.0 (TID 83) in 292 ms on 100.96.9.14 (84/100)
17/03/08 22:35:40 INFO TaskSetManager: Starting task 85.0 in stage 0.0 (TID 85, 100.96.9.14, partition 85, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:40 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 85 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:40 INFO TaskSetManager: Finished task 84.0 in stage 0.0 (TID 84) in 235 ms on 100.96.9.14 (85/100)
17/03/08 22:35:40 INFO TaskSetManager: Starting task 86.0 in stage 0.0 (TID 86, 100.96.9.14, partition 86, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:40 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 86 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:40 INFO TaskSetManager: Finished task 85.0 in stage 0.0 (TID 85) in 160 ms on 100.96.9.14 (86/100)
17/03/08 22:35:40 INFO TaskSetManager: Starting task 87.0 in stage 0.0 (TID 87, 100.96.9.14, partition 87, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:40 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 87 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:40 INFO TaskSetManager: Finished task 86.0 in stage 0.0 (TID 86) in 199 ms on 100.96.9.14 (87/100)
17/03/08 22:35:40 INFO TaskSetManager: Starting task 88.0 in stage 0.0 (TID 88, 100.96.9.14, partition 88, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:40 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 88 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:40 INFO TaskSetManager: Finished task 87.0 in stage 0.0 (TID 87) in 170 ms on 100.96.9.14 (88/100)
17/03/08 22:35:40 INFO TaskSetManager: Starting task 89.0 in stage 0.0 (TID 89, 100.96.9.14, partition 89, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:40 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 89 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:40 INFO TaskSetManager: Finished task 88.0 in stage 0.0 (TID 88) in 139 ms on 100.96.9.14 (89/100)
17/03/08 22:35:41 INFO TaskSetManager: Starting task 90.0 in stage 0.0 (TID 90, 100.96.9.14, partition 90, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:41 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 90 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:41 INFO TaskSetManager: Finished task 89.0 in stage 0.0 (TID 89) in 181 ms on 100.96.9.14 (90/100)
17/03/08 22:35:41 INFO TaskSetManager: Starting task 91.0 in stage 0.0 (TID 91, 100.96.9.14, partition 91, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:41 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 91 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:41 INFO TaskSetManager: Finished task 90.0 in stage 0.0 (TID 90) in 177 ms on 100.96.9.14 (91/100)
17/03/08 22:35:41 INFO TaskSetManager: Starting task 92.0 in stage 0.0 (TID 92, 100.96.9.14, partition 92, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:41 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 92 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:41 INFO TaskSetManager: Finished task 91.0 in stage 0.0 (TID 91) in 150 ms on 100.96.9.14 (92/100)
17/03/08 22:35:41 INFO TaskSetManager: Starting task 93.0 in stage 0.0 (TID 93, 100.96.9.14, partition 93, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:41 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 93 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:41 INFO TaskSetManager: Finished task 92.0 in stage 0.0 (TID 92) in 241 ms on 100.96.9.14 (93/100)
17/03/08 22:35:41 INFO TaskSetManager: Starting task 94.0 in stage 0.0 (TID 94, 100.96.9.14, partition 94, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:41 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 94 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:41 INFO TaskSetManager: Finished task 93.0 in stage 0.0 (TID 93) in 198 ms on 100.96.9.14 (94/100)
17/03/08 22:35:42 INFO TaskSetManager: Starting task 95.0 in stage 0.0 (TID 95, 100.96.9.14, partition 95, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:42 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 95 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:42 INFO TaskSetManager: Finished task 94.0 in stage 0.0 (TID 94) in 165 ms on 100.96.9.14 (95/100)
17/03/08 22:35:42 INFO TaskSetManager: Starting task 96.0 in stage 0.0 (TID 96, 100.96.9.14, partition 96, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:42 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 96 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:42 INFO TaskSetManager: Finished task 95.0 in stage 0.0 (TID 95) in 180 ms on 100.96.9.14 (96/100)
17/03/08 22:35:42 INFO TaskSetManager: Starting task 97.0 in stage 0.0 (TID 97, 100.96.9.14, partition 97, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:42 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 97 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:42 INFO TaskSetManager: Finished task 96.0 in stage 0.0 (TID 96) in 156 ms on 100.96.9.14 (97/100)
17/03/08 22:35:42 INFO TaskSetManager: Starting task 98.0 in stage 0.0 (TID 98, 100.96.9.14, partition 98, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:42 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 98 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:42 INFO TaskSetManager: Finished task 97.0 in stage 0.0 (TID 97) in 181 ms on 100.96.9.14 (98/100)
17/03/08 22:35:42 INFO TaskSetManager: Starting task 99.0 in stage 0.0 (TID 99, 100.96.9.14, partition 99, PROCESS_LOCAL, 5525 bytes)
17/03/08 22:35:42 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 99 on executor id: 0 hostname: 100.96.9.14.
17/03/08 22:35:42 INFO TaskSetManager: Finished task 98.0 in stage 0.0 (TID 98) in 199 ms on 100.96.9.14 (99/100)
17/03/08 22:35:42 INFO TaskSetManager: Finished task 99.0 in stage 0.0 (TID 99) in 185 ms on 100.96.9.14 (100/100)
17/03/08 22:35:42 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/03/08 22:35:42 INFO DAGScheduler: ResultStage 0 (reduce at /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py:22) finished in 21.845 s
17/03/08 22:35:42 INFO DAGScheduler: Job 0 finished: reduce at /root/volumes/source.ml/jupyterhub.ml/scripts/spark/pyspark_pi.py:22, took 22.450788 s
Pi is roughly 3.147120
17/03/08 22:35:43 INFO SparkUI: Stopped Spark web UI at http://100.96.9.57:4041
17/03/08 22:35:43 INFO StandaloneSchedulerBackend: Shutting down all executors
17/03/08 22:35:43 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
17/03/08 22:35:43 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/03/08 22:35:43 INFO MemoryStore: MemoryStore cleared
17/03/08 22:35:43 INFO BlockManager: BlockManager stopped
17/03/08 22:35:43 INFO BlockManagerMaster: BlockManagerMaster stopped
17/03/08 22:35:43 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/03/08 22:35:43 INFO SparkContext: Successfully stopped SparkContext
17/03/08 22:35:44 INFO ShutdownHookManager: Shutdown hook called
17/03/08 22:35:44 INFO ShutdownHookManager: Deleting directory /tmp/spark-2f7edac3-6d2a-438f-ac46-8302914167b2
17/03/08 22:35:44 INFO ShutdownHookManager: Deleting directory /tmp/spark-2f7edac3-6d2a-438f-ac46-8302914167b2/pyspark-1f61d332-b9f3-45fb-ae2d-31b5c7eece5c
In [ ]:
Content source: Resly/pipeline
Similar notebooks: