ANOMALY DETECTION FRAMEWORK FOR BIG DATA

Outline Background Problem Related Work Result Introduction Methodology Implementation Result Conclusions/Future Work by NKG

Introduction The volume of data hitting the servers of financial institutions is increasing at an exponential rate. This makes it difficult for their analysis Big Data Anomaly Detection Framework Spark: Spark Mlib, Support Vector Machine, was used to build a model to detect Fraudulent Transaction. by NKG

Background What is Big Data? The iBig data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. Anomaly Detection: Anomaly detection is defined as the task of finding instances in a dataset which are different from the norm. (Goldstein, 2014). The idea of Big data was born out of the fact that considering the growing volume of data which are large and complex, traditional tools are no longer sufficient to process this data. In this regard, predictive modeling such as anomaly detection, should be performed on this Big data. by NKG

Background Cont. What is Data mining : Data mining is a process used by companies to turn raw data into useful information. By using software to look for patterns in large batches of data, businesses can learn more about their customers and develop more effective marketing strategies as well as increase sales and decrease costs. Credit card fraud is a wide-ranging term for theft and fraud committed using or involving a payment card, such as a credit card or debit card, as a fraudulent source of funds in a transaction. (Wikipedia, 2016) Data mining is a powerful new technology with great potential to help companies focus on the most important information in the data they have collected about the behavior of their customers and potential customers. by NKG

Problem With this exciting growth comes a new challenge whereby, new products and a growing customer base translate to increased pressure on banks to manage and secure their data. The problem under study is to look at ways of best algorithm and techniques to detect fraud (data anomaly) in financial institutions based on online banking transactions (using Credit Card) to improve decisions. By the nature of their businesses, financial firms are required to store a large volume of data under incredibly strict compliance with a number of regulators. by NKG

Introduction Cont. General Objectives Review the concept of big data, tools and techniques for real time data analysis. Develop a model for anomaly detection Propose a framework based on adversarial machine learning techniques to detect anomaly in real time environment in big data by NKG

Related Work Data Mining Process Data Clustering and Integration Selection and Transformation Data Mining Interpretation/Evaluation Knowledge by NKG

Big Data Related Work Cont. Volume Velocity Variety Veracity Value Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. by NKG

Technologies of Big Data Related Work Cont. Technologies of Big Data There are a lot of big data technologies, Examples are Spark Hadoop Cassandra Mongo DB No SQL by NKG

Related Work Cont. Mahest et al. (2015), using classification profiling of the input codes to differentiates between good and malicious codes. The Intelligent Customer Analytics for Recognition and Exploration (iCARE), a framework presented to analyze customer behavior using banking big data. which was leverage on IBM products of SPSS Analytic Server and InforSphere. Credit Card Fraud Detection using SVM and reduction of False Alarms by NKG

Compering SVM-M with iCARE Related Work Cont. Compering SVM-M with iCARE SVM-S iCARE Distributing of Data Real time Streaming Spark Apache Runs in Memory Distributing of Data Not real time streaming IBM InforSphere Runs in External Storage by NKG

Methodology Experiment was conducted to solve the problem using Spark Apache big data Platform. Support Vector Machine with Spark (SVM-S) Framework was used. Data for the experiment was credit card transactions. by NKG

Methodology Cont. Data Sources Client Creating Files Bank Data Sources by NKG

SVM-S Model The Support Vector Machine with Spark (SVM-S) Model combine both historical and streaming data for prediction. It can stream real time transactions for predicting anomalies with the train data by NKG

SVM-S Architecture The SVM-S Architecture combine both Support Vector Machine and Logistic Regression for training the data model. It stand on HDFS for acquiring data for prediction. by NKG

SVM-S Framework The SVM-S Framework It stream data at real time using the Spark streaming. The SVM-S Framework process supervised, semi-supervised and unsupervised data. by NKG

Implementation Set up your Spark Create path for Python or Scala Programming the data used was from literature review, credit card transaction of German companies and Australia agencies  the data was .json file, and it was analyze using spark stand alone on i5 laptop with 8 Gig of RAM.  by NKG

Implementation Data Credit card transactions was use for implementation of the SVM-S Framework. Normal transactions was mixed with anomaly transactions to be detected by SVM-S Framework by NKG

Result SVM-S was very accurate and robust to detect injected fraud transactions. SVM-S Framework is very efficient when compare with iCARE Framework. by NKG

Conclusion This thesis has considered how big data can be leveraged in the financial sector. In particular, the study has presented an approach to anomaly detection in big data. This thesis proposed Anomaly Detection Framework to enhance fraud detection in financial institutions. To cope with the velocity, veracity and volume of Big Data, the anomaly detection framework relies on a fast, albeit less accurate and live streaming of data to find anomalies in real-time. This is one of a kind to be introduced by the financial institutions in Ghana. by NKG

Reference by NKG

Future Work It is important to consider the other common types of application, datasets. However, adding new features to improve the framework to serve other data types like sensors. Secondly, to implement the framework, in a real business environment, the framework could also be feasible with decision making systems within such an environment. Again, the component of ADF, allows further components to be added or upgraded. However, perhaps research will be presented that shows different machine learning algorithms to perform better in implementations. by NKG