Institutional Repository
Technical University of Crete
Technical University of Crete
EN
|
EL
| URI: | http://purl.tuc.gr/dl/dias/3FE9BF78-267D-4112-97BA-1E5731348359 | ||
| Year | 2020 | ||
| Type of Item | Diploma Work | ||
| License |
|
||
| Bibliographic Citation | Ioannis Baikousis, "Migration state among jobs in Apache Flink", Diploma Work, School of Electrical and Computer Engineering, Technical University of Crete, Chania, Greece, 2020 https://doi.org/10.26233/heallink.tuc.87679 | ||
| Appears in Collections |
On a daily basis more and more data is produced and needs to beprocessed to extract useful information. Specifically, processing datastreams is vital and requires high-performance resources to query them inreal time. Big Data processing frameworks have been developed to handleefficient complex queries on data streams. As the amount of data expands,the frameworks have to adapt to processing requirements, thus it is essentialto support updates and upgrades both in hardware and softwareinfrastructures over time. Therefore, migration mechanisms have beendeveloped in order to give the ability to frameworks to evolve, guaranteeingno data losses.In this diploma thesis we provide a migration algorithm in Apache Flinkwhich gives you the opportunity to manage many operator states amongdifferent Flink jobs and submit them into another cluster with no data losses.Additionally,it enables us to merge, split or rescale jobs in order to adapt toprocessing requirements. Our algorithm is based on the State Processor APIwhich is provided by Flink and it is implemented on RapidMiner Studio whichgives us the ability to design workflow easily and quickly.To validate our approach, we designed some workflows using simpleoperators on RapidMiner studio and present a complete detailed cluster-mode execution with many test-cases as merging and splitting the workflowsand migrating the state without data losses proving the correctness of ourmigration algorithm.
Copyright © DIAS 2013
