linux 网络状态命令 详解linux运维工程师高级篇（大数据安全方向）(9)

上传spark文件到opt

# hdfs dfs -mkdir -p /tmp/sparkwordcount/
# hdfs dfs -mkdir -p /tmp/sparkwordcount/input
# hdfs dfs -put /opt/sparkwordcountinput.txt /tmp/sparkwordcount/input
# hdfs dfs -put /opt/spark_word_count.jar  /tmp/sparkwordcount/

#spark测试文件

sparkwordcountinput.txt  spark_word_count.jar

spark命令提交任务

 # spark-submit   --class com.vrv.bigdata.ml.DataExtract2   --master yarn   --deploy-mode cluster   --principal spark-test@CESHI.COM   --keytab /etc/security/keytabs/spark.headless.keytab   hdfs://hdp140:8020/tmp/sparkwordcount/spark_word_count.jar   hdfs://hdp140:8020/tmp/sparkwordcount/input   hdfs://hdp140:8020/tmp/sparkwordcount/output/spark_work_count
17/10/31 01:15:28 INFO Client:
 client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
 diagnostics: N/A
 ApplicationMaster host: 192.168.2.143
 ApplicationMaster RPC port: 0
 queue: default
 start time: 1509383715631
 final status: SUCCEEDED
 tracking URL: http://hdp141:8088/proxy/application_1509379053332_0014/
 user: spark
17/10/31 01:15:28 INFO ShutdownHookManager: Shutdown hook called
17/10/31 01:15:28 INFO ShutdownHookManager: Deleting directory /tmp/spark-40e868df-ca58-4389-b20c-03d2717516cc

本文来自电脑杂谈，转载请注明本文网址：
http://www.pc-fly.com/a/jisuanjixue/article-66746-9.html