Download presentation
Presentation is loading. Please wait.
1
A TROPOS Comp 415 Project In partnership with JPMorganChase
2
I NTRODUCTORY R EMARKS COMP 415 Class: Brad Chelsea Dave Felipe Hubert Matt Sohum Dr. Wong Jacy Grannis
3
A W ORD F ROM O UR S PONSORS... Jacy Grannis has been our customer contact within JPMorganChase. He is a Rice Computer Science graduate and works in JPMorgan’s Houston office.
4
COMP 415—S UMMARY Software engineering Real customer, real experience Team environment Communication Organization
5
COMP 415—T HE BENEFITS Team programming experience is sought by employers and not often found in undergraduates Saves on-the-job learning time Industry exposure
6
COMP 415—T HE SEQUENCE COMP 410 COMP 415 Teaching assistant for both courses Need more courses
7
A TROPOS …semester in review
8
A TROPOS —S UCCESS Correlation Network View Single Edge View Remote communication
9
A TROPOS —M ISTAKES Teams change Individuals change Team structure Small team Proxy pattern Integration
10
S YSTEM R EQUIREMENTS …an Atropos primer
11
S YSTEM R EQUIREMENTS —C ORRELATION Track messages as they travel between servers Different systems have different message formats Messages have no fixed unique ID Servers may split and merge messages
12
S YSTEM R EQUIREMENTS —S CALABILITY System must handle thousands of messages per second Components may run on separate machines Concurrency Asynchronous communications
13
S YSTEM R EQUIREMENTS —R ECOVERABILITY Failure of one component should not bring the system down Failure detection Component restart Restore component state Replay dropped messages
14
S YSTEM D ESIGN …the view from the top
15
Distributed system Service-oriented architecture (SOA) Component registration A TROPOS S YSTEM D ESIGN —U NREGISTERED
16
Distributed system Service-oriented architecture (SOA) Component registration A TROPOS S YSTEM D ESIGN —R EGISTERED
17
D ESIGN —A BSTRACTION AND THE P ROXY P ATTERN Allows flexible system configuration Beneficial for testing purposes and agile development Unbiased by transfer protocol
18
D ESIGN —D YNAMIC C ORRELATION Concepts of finding matches from message organization are logically distinct Decouple algorithm from data Correlation engines organize messages and use strategies to fine matches Changing strategies at runtime enables intelligent load balancing and independence from JPM system configuration
19
A TROPOS D EMO …which will totally work
20
S TART C ONTROLLER AND R OUTER Initialize the controller Service that manages the system Router registers with controller
21
S TART C ORRELATION E NGINE, D ATA C LIENT Correlation Engines and Data Clients run in their own JVMs On startup, they register with the controller Controller configures them to connect them into the system
22
S TARTING V IEW S YSTEM View Provider View Client
23
C ONNECT AND S TART C LIENT View Client connects to system Requests data
24
C HANGING S OURCES AND E DGES Utilize IAdmin API to add sources Dynamically add and remove edges Check system status after updates
25
S INGLE E DGE L ATENCY V IEW Access Dialog Clicking edge in Dashboard View Display Time-chunk average of latency Scalable to different time periods Static or updatable
26
C USTOM L ATENCY V IEW See correlation during a specific time window
27
U NCORRELATED M ESSAGES Messages change, machine connections change, parts of the topography may disappear completely Messages will stop correlating View uncorrelated messages over a period of time Updates live (currently every 30 secs)
28
D ISCONNECT V IEW C LIENT Disconnect a view client currently registered View client is unregistered Future updates are no longer sent
29
R EMOTE C OMMUNICATION …bridging gaps
30
R EMOTE C OMMUNICATION No guarantee that components are on the same machine Need a way to communicate between multiple components Solution must be Scalable Flexible Pluggable
31
M ULE —B ENEFITS Provides a scalable package solution Open source Provides a flexible framework that can adapt to any architecture Can tie into many messaging protocols
32
M ULE —H ARDSHIPS Little support and documentation Remote method invocation can cause problems Needs to be scaled to work effectively
33
F UNCTIONAL C OMMUNICATION Proxy pattern connects components Controller as an object on the Mule server Mule support behind remote calls abstracted out
34
D ATA C LIENT …the beginning of it all
35
D ATA C LIENT —G OALS Vehicle for getting messages into the system Robust Abstract Recoverable Extensible
36
D ATA C LIENT —M ESSAGE S OURCES Entry point for messages Accepts messages with a simple API accept(message)... too easy? Next step: router
37
D ATA C LIENT —A BSTRACT Messages can come from anywhere Log parsers Applications producing the messages Manual API calls Abstracting out information about the source is crucial for interoperability
38
D ATA C LIENT —R ECOVERABILITY Router dies... What happens to messages that have been sent through the Data Client but are dropped? Take a lesson from Tivo! Store information about messages that have been seen and replay! Controller detects failure in the system Can ynchronize all the modules in the system by replaying the Data Client
39
D ATA C LIENT —E XTENSIBILITY Multiple sources can send information to one data client Require that incoming messages are be tagged with their source ID Data Clients can send to other Data Clients Available for common special processing that can be added at run time Added and removed via registration at run time with the Controller
40
R OUTER …the grunt work
41
R OUTER —R OLE Data Clients are only aware of Router All messages simply forwarded Router responsible for mapping Data Clients to Correlation Engines Data Client to Correlation Engine mapping not 1:1 One-to-many relationship
42
R OUTER —E XTENSIBILITY Routers can make use of the composite pattern Updated dynamically at runtime by the Controller Whenever Data Clients or Correlation Engines register (or unregister), the router must reflect those changes
43
C ORRELATION E NGINE …the workhorse
44
C ORRELATION E NGINE —R OLE Organizes messages received Applies correct strategies between various sources, as determined by the system’s IAdmin Writes correlation information to a memory store Keeps a store of what messages have been received for replay purposes
45
C ORRELATION E NGINE —D ESIGN C ONCERNS Configurability Should handle addition of sources and new rules at runtime Concurrency Thread safety Use of concurrency to take advantage of multi-cores
46
C ORRELATION E NGINE —I MPLEMENTATION D ETAILS Makes messages appear to have been received in order; facilitates strategy implementation Configurable correlation strategies Handle rules for correlation Insulated from issues related to system configuration, message order, etc.
47
U SER I NTERFACE …the pretty side
48
V IEW C LIENT —T ECHNOLOGIES Eclipse RCP Application Java Universal Network/Graph (JUNG) framework JFreeChart framework
49
E CLIPSE RCP A PPLICATION —P OSITIVES Portability Integration with current JPMorgan systems Faster development on Eclipse IDE Powerful framework simplifies component connections
50
E CLIPSE RCP A PPLICATION —N EGATIVES SWT less popular than AWT Fewer resources for help Steep learning curve
51
V IEW C LIENT —JUNG F RAMEWORK Ability to graph nodes based on set of vertices and edges Robust decorator-based design for great customization Interactive display with the graph
52
V IEW C LIENT —JF REE C HART F RAMEWORK SWT-native chart creation package Wide variety of charts supported by default Interactive graph display Intuitive package to work with as a developer
53
V IEW P ROVIDER Serves data as requested by many concurrent view clients Asynchronous "pull" data delivery system Designed to be able to live on a different server than the view client Ability to handle several different types of data requests Design allows several View Providers to be active, and for clients to connect to specific ones
54
V IEW P ROVIDER —N ETWORK T OPOLOGY Provides the edges and vertices that comprise the topology Pulls data from the PostgreSql datastore and converts to viewable form
55
V IEW P ROVIDER —S INGLE E DGE L ATENCY Provides time series of latencies between two vertices Parameterization of factors such as time range
56
V IEW P ROVIDER —U NCORRELATED M ESSAGES Provides list of messages that have not been correlated by the system Parameterization of time range allows specific messages to be identified
57
V IEW P ROVIDER —S INGLE M ESSAGE Provides data about the computed correlated paths of specified messages Includes information about the latency at each vertex in a given path
58
T HANK YOU FOR COMING ! …special thank you to Dr. Wong, Jacy Grannis, and JPMorganChase for making this semester possible!
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.