Institutional Repository
Technical University of Crete
Technical University of Crete
EN
|
EL
| URI | http://purl.tuc.gr/dl/dias/6D2F87E6-508B-4687-97B1-9A5CA530ACB6 | - |
| Identifier | https://doi.org/10.26233/heallink.tuc.70651 | - |
| Language | en | - |
| Extent | 48 pages | en |
| Title | Scaling text processing pipelines using Apache Spark | en |
| Creator | Katsani Merieme | en |
| Creator | Κατσανι Μεριεμε | el |
| Contributor [Thesis Supervisor] | Deligiannakis Antonios | en |
| Contributor [Thesis Supervisor] | Δεληγιαννακης Αντωνιος | el |
| Contributor [Committee Member] | Garofalakis Minos | en |
| Contributor [Committee Member] | Γαροφαλακης Μινως | el |
| Contributor [Committee Member] | Lagoudakis Michail | en |
| Contributor [Committee Member] | Λαγουδακης Μιχαηλ | el |
| Publisher | Πολυτεχνείο Κρήτης | el |
| Publisher | Technical University of Crete | en |
| Academic Unit | Technical University of Crete::School of Electrical and Computer Engineering | en |
| Academic Unit | Πολυτεχνείο Κρήτης::Σχολή Ηλεκτρολόγων Μηχανικών και Μηχανικών Υπολογιστών | el |
| Content Summary | Big data, which is derived from humans or machines, starting with social media and extending to smartphones or sensors, in forms of texts, images or transactions, is a continuously evolving field. Thus, the ongoing increase of data generated creates a need for knowledge extraction from it, through data analysis. Several areas are engaged in data mining, and in particular the area of machine learning which has been well established over the past years. Various techniques and methods of machine learning are trying to solve big data problems and these two areas consist now an integral part. This particular combination is the main subject of this study, which aims to implement a large-scale text processing architecture. More specifically, this architecture focuses on processing streaming texts derived from Reddit in real-time and the classification thereof as sarcastic or non-sarcastic through a machine learning model. The architecture uses the latest technologies in the field of information processing through distributed platforms such as Apache Kafka and Spark as well as state-of-the-art but also simple and powerful ML algorithms, i.e Random Forests, Naive Bayes and Logistic Regression. After comparing the methodology and design of each individual piece forming the final layout, a selection of the most appropriate model is made followed by the implementation of the framework. Success rates exported were quite close to the relevant literature and sometimes higher, depending on each technique examined. Finally, results are indexed in the distributed search engine Elasticsearch and are evaluated through the Kibana plugin. | en |
| Type of Item | Διπλωματική Εργασία | el |
| Type of Item | Diploma Work | en |
| License | http://creativecommons.org/licenses/by/4.0/ | en |
| Date of Item | 2018-01-02 | - |
| Date of Publication | 2017 | - |
| Subject | Big data | en |
| Subject | Machine learning | en |
| Subject | NLP | en |
| Subject | Natural Language Processing | en |
| Bibliographic Citation | Merieme Katsani, "Scaling text processing pipelines using Apache Spark", Diploma Work, School of Electrical and Computer Engineering, Technical University of Crete, Chania, Greece, 2017 | en |
Copyright © DIAS 2013
