Spark Structured Streaming的一些link

最近在测试相关内容, 以下列了一些我觉得有用的link

官方文档

Spark Structured Streaming Programming Guide

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html

https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html



databricks Structured Streaming

https://docs.databricks.com/spark/latest/structured-streaming/index.html

databricks上的一个汇总link

Anthology of Technical Assets on Apache Spark’s Structured Streaming

https://databricks.com/blog/2017/08/24/anthology-of-technical-assets-on-apache-sparks-structured-streaming.html

里面

Spark Structured Streaming A new high-level API for streaming

https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html

Real-time Streaming ETL with Structured Streaming in Apache Spark 2.1 共5篇

https://databricks.com/blog/2017/01/19/real-time-streaming-etl-structured-streaming-apache-spark-2-1.html

https://databricks.com/blog/2017/02/23/working-complex-data-formats-structured-streaming-apache-spark-2-1.html

https://databricks.com/blog/2017/04/26/processing-data-in-apache-kafka-with-structured-streaming-in-apache-spark-2-2.html

https://databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-structured-streaming.html

https://databricks.com/blog/2017/05/18/taking-apache-sparks-structured-structured-streaming-to-production.html

CSDN上的一篇

Databrick 's Blog on Spark Structured Streaming Summary

https://blog.csdn.net/asd136912/article/details/82147657

系统示例测试

目前测试2.3.0运行自带的structured_kafka_wordcount.py会报错

bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0  examples/src/main/python/sql/streaming/structured_kafka_wordcount.py     testspark:9093 subscribe test1 

错误信息

py4j.protocol.Py4JJavaError: An error occurred while calling o32.load.

: org.apache.kafka.common.config.ConfigException: Missing required configuration "partition.assignment.strategy" which has no default value.

        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)

2.2.2和2.3.2都可以

本文由seng 创作，采用 知识共享署名-相同方式共享 3.0 中国大陆许可协议 进行许可。

转载、引用前需联系作者，并署名作者且注明文章出处。

本站文章版权归原作者及原出处所有 。内容为作者个人观点， 并不代表本站赞同其观点和对其真实性负责。本站是一个个人学习交流的平台，并不用于任何商业目的，如果有任何问题，请及时联系我们，我们将根据著作权人的要求，立即更正或者删除有关内容。本站拥有对此声明的最终解释权。