Applications for Big Data processing and Data-driven Decision Support using Spark and Machine Learning

Dependencies

• Java version: 8
• Apache Spark: 3.0.0
• Apache Hadoop: 3.3.0
• Apache Maven: 3.3.0

Technology stack

• IntelliJ IDEA Ultimate

Features

• Spark: spark-core_2.12, spark-sql_2.12
• build plugins: maven-compiler-plugin, maven-assembly-plugin
• CSV, SparkConf, JavaSparkContext, JavaRDD, parallelize, BufferedReader, InputStream, InputStreamReader, HashSet, Set, List, ArrayList, Arrays, ImmutableList, Serializable, flatMap, filter, collect, map, flatMap, reduce, mapToPair, reduceByKey, sortByKey, take, contains, ...