Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing Lycom Dmitri 1, Trzasala Milena 2, Students Gr. 1 -8BM23, 2 –AO Tomsk Polytechnic University, Russia 2 -Wroclaw University of Technology, Poland Scientific advisor: Dr. Sergey Axyonov

Gaia project Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing 2 European Space Agency (ESA) mission Launch Date: 2013 Mission End: after 5 years (2018) Launch Vehicle: Soyuz-Fregat Orbit: Lissajous-type orbit around L2 OBJECTIVES: Gaia is an ambitious mission to chart a three-dimensional map of our Galaxy, the Milky Way, in the process revealing the composition, formation and evolution of the Galaxy. MISSION: Produce a stereoscopic and kinematic census of about one billion stars in our Galaxy and throughout the Local Group. Detection and orbital classification of tens of thousands of extra-solar planetary systems.

Objective & Tasks 3 Motivation  Presence of large-scale datasets from the Gaia space project  Need for fast and accurate analysis of received information  The technique can be used to process other huge industrial databases Objective High Performance of Vectors Clustering Techniques for Rapid and Reliable ESA Datasets Analysis Tasks  Design a technique that allows to get effective data distribution and supercomputer processing.  Create a program implementation of the suggested technique that is capable to be used with computational cluster.  Test and estimate the performance of developed software with ESA testing datasets. Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing

Datasets & Attributes 4 Analyzing Stars Features Testing Datasets Attribute NameMeaning log-f1 Log of the first frequency log-f2 Log of the second frequency log-af1h1-t Log amplitude, first harmonic, first frequency log-af1h2-t Log amplitude, second harmonic, first frequency log-af1h3-t Log amplitude, third harmonic, first frequency log-af1h4-t Log amplitude, fourth harmonic, first frequency log-af2h1-t Log amplitude, first harmonic, second frequency log-af2h2-t Log amplitude, second harmonic, second frequency log-crf10 Amplitude ratio between harmonics of the first frequency pdf12 Phase difference between harmonics of first frequency Varrat Variance ratio before and after first frequency subtraction B-V Colour index V-I Colour index Dataset# Instances# AttributesUsage Synth43K Validation of clustering quality and implementation of algorithm Synth Scalability performance testing Synth Scalability performance testing Synth Scalability performance testing Synth Scalability performance testing Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing

DBSCAN Clustering technique 5 Allows to get clusters (groups of points) with any shape. Based on the points density Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing Disadvantage: Performance!!! Heavily Depends on Dataset Sizes

Data Distribution Problem 6 Each process that is connected with a computational node gets just a part of data. Processes can’t detect clusters because neighboring points are located in different memory space Our Solution Message Passing Interface MPI_Scatter result Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing

Gustaffson-Kessel Fuzzy Clustering 7 Allows to generate clusters with hyperellipse shape ⇒ more natural clustering. Computes fuzzy matrix that reflects distances between vectors and centers based on points density. ⇒ arrangement of points for indexing and boundary points detection Base Computations of clustering are Matrix operations ⇒ easy to parallize Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing

Preclustering 8 Use a part of data to clusterize ⇒ speed up processing. Use some computational nodes with different data ⇒ reliability of analysis. Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing

The Most General Clustering Search 9 Detection the most effective splitting in terms of density and distances between vectors. This distribution reflects the nature of data. Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing

Clusters join (Output Synchronization) 10 Each point has a fuzzy vector that reflects distances between the point and centers of clusters. Perform “Join” operation if distances between boundary points in two processes are less the radius of neighborhood. We need just a very small part of clustered points that are located at the borders between clusters ⇒ speedup. Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing

Implementation 11 The program is developed with MS Visual C# (multi-core + multi-node computations were used) Libraries of Parallel Computing.NET4 (Multi-core processing) Matrix operations in GK clustering Cluster equivalence (The Most General Clustering search) Libraries of Distributed Programming MPI.NET (Multi-node processing) Data Scattering (for Preclustering) Data Distribution (for DBSCAN Procedure) Data Broadcasting (The Most General Clustering sending) Clusters join (Point-to-Point Communication) The program was tested with SKIF-TPU Computational Cluster (used 20 computational nodes, 80 processors) Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing

Performance 12 Performance of the suggested technique to analyze the Synth-10 5 Dataset in distributed manner. ⇒ The approach really increases the performance of algorithm. Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing

Results 13 Designed a new original approach to parallel- distributed large-scale dataset clustering Created a C# program that implements the suggested model based on MPI.NET and parallel libraries of.NET 4. The software was tested to estimate the performance of suggested approach with the Gaia test datasets The suggested technique increases the DBSCAN performance. Parallel-distributed processing is much more effective versus standard DBSCAN method. Usage of Parallel Fuzzy Clustering for Performance of Distributed ESA Datasets Processing