Applications for Big Data using Spark, Deep Learning and Natural Language Processing methods

Technology stack & dependencies

• Java: 8
• Apache Spark: 3.0.0
• Hadoop: 3.3.0
• Apache Maven: 3.3.0
• Deeplearning4j
• OpenNLP
• Python: 3.9.1
• IntelliJ IDEA Ultimate

Features

• build plugins: maven-compiler-plugin, maven-assembly-plugin
• Spark: spark-core_2.12, spark-sql_2.12
• CSV, SparkConf, JavaSparkContext, JavaRDD, parallelize, BufferedReader, InputStream, InputStreamReader, HashSet, Set, List, ArrayList, Arrays, ImmutableList, Serializable, flatMap, filter, collect, map, flatMap, reduce, mapToPair, reduceByKey, sortByKey, take, contains, ...
• deeplearning4j-core
• CSVRecordReader, FileSplit, Evaluation, MultiLayerConfiguration, NeuralNetConfiguration, DenseLayer, OutputLayer, MultiLayerNetwork, DataSet, DataNormalization, NormalizerStandardize, LossFunctions, HashMap, Map, ...
• SentenceDetection(), Tokenize(), findName(), POSTag(), chunk(), Parse(), ...
• ChunkerME, ChunkerModel, PerformanceMonitor, ParserTool, POSModelLoader, NameFinderME, TokenNameFinderModel, Parse, ParserFactory, ParserModel, POSModel, POSSample, POSTaggerME, SentenceDetectorME, SentenceModel, Tokenizer, TokenizerME, TokenizerModel, WhitespaceTokenizer, FileInputStream, ...
• Java Embedded Python