Avro Apache Course: Distributed class Student ID: AM Name: Azzaya Galbazar

Overview-What is Avro?  Avro is an Apache open source project that provides two services for the Hadoop(data serialization and exchange). Avro is recent serialization system.  Interoperability Can Serialize into Avro/Binary or Avro/JSON Supports reading and writing protobufs and thrift

Overview-Avro provides..?  Rich data structures with schema designed over JSON A compact, fast binary format. A container file, to store persistent data. Remote procedure call (RPC). Simple integration with dynamic languages.  Code generation is not required to read or write data files nor to use or implement RPC protocols.  Code generation as an optional optimization, only worth implementing for statically typed languages.

Overview  Avro uses JSON for Interface Description Language(IDL) To specify data types To specify protocols  Review: JavaScript Object Notation is just a light- weight text-based standard for data interchange.

Overview-Why the need for Avro?  Primary usage in Hadoop, provides standard:  Serialization format for persistent data  Wire format for communication Among Hadoop nodes. From client programs to Hadoop services.

Overview  Avro relies on schemas.  Schema stored with data  Each datum written with no per-value overheads. Thus serialization is fast and small  Avro in RPC:  Schema exchange during client-server handshake  Correspondence in fields can be easily resolved.

Overview-APIs  Supporting API for:  Java  C  C++  C#  Python  Ruby

Specification  A Schema is represented in JSON by on of:  A JSON string, naming a defined type.  A JSON object, of the form: {“type”: ”type name” …attributes…}  A JSON array, representing a union of embedded types.  Primitive types: null, boolean, int, long, float, double, bytes, string  Complex types: records, enums, arrays, maps, unions, fixed

Apache Avro with Maven Java 1. Apache Maven is a software project management and comprehension tool. 1. Based on the concept of a project object model (POM), 2. Maven can manage a project's build, reporting and documentation from a central piece of information

Apache Avro with Maven Java 1.Add two dependencies to pom.xml-the one is Apache Avro library, the other one is maven plugin that allows us to generate Java classes.

Apache Avro with Maven Java 1.Add two dependencies to pom.xml-the one is Apache Avro library, the other one is maven plugin that allows us to generate Java classes.

Apache Avro with Maven Java 2.Defining a schema #a schema file can only contain a single schema definition.

Apache Avro with Maven Java 2.Serializing and deserializing from a File # serializes book to file and deserializes it and print it to output.

Apache Avro with Maven Java 2.Serializing and deserializing from a File # serializes book to file and deserializes it and print it to output.

Apache Avro with Maven 2.Describing functions #DataFileWriter converts Java object into an in-memory serialized format. #SpecificDatumWriter extracts the schema from specified type. #DataFileWriter writes the serialized record, as well as the schema.

Apache Avro with Maven Java 4.Running the example code 5.Result output.

Thank you for your attention