Big Data Frameworks

A comparition between Hadoop, Apache Spark, Apache Flink, Apache Storm and Apache Sazam

Author: Marco Olimpio - marco.olimpio at gmail

Objetives: The purpose of this notebook is to make a comparition between Hadoop, Apache Spark, Apache Flink, Apache Storm and Apache Sazam frameworks. Given that these technologies for 'Big Data'are meant to make more than one think (like Spark) we will considerate mode than one scenario for the comparition like streaming processing, machine learning implementations and so on.

Content:

1. Hadoop
2. Apache Spark
3. Apache Flink
4. Apache Storm
5. Apache Sazam
6. Comparitions
   1. MapReduce and Spark
   2. Streaming Processing
   3. Implementation
   4. Best suited for...
7. Conclusions
8. References

Apache Kafka Apache Apex

1. Hadoop

2. Apache Spark

3. Apache Flink

4. Apache Storm

5. Apache Sazam

6. Comparitions

7. Conclusions

8. References

• Streams
  • https://developer.twitter.com/en/docs
  • Apache Kafka
  • https://ci.apache.org/projects/flink/flink-docs-release-1.3/
  • https://www.digitalocean.com/community/tutorials/hadoop-storm-samza-spark-and-flink-big-data-frameworks-compared
• Misc.
  • https://www.cloudera.com/more/news-and-blogs/press-releases/2016-11-07-apache-spark-survey-reveals-increased-growth-in-users-and-new-workloads.html