Presentation is loading. Please wait.

Presentation is loading. Please wait.

The Clarens Web Service Framework Frank van Lingen (Caltech) on behalf of the Clarens developers.

Similar presentations

Presentation on theme: "The Clarens Web Service Framework Frank van Lingen (Caltech) on behalf of the Clarens developers."— Presentation transcript:

1 The Clarens Web Service Framework Frank van Lingen (Caltech) on behalf of the Clarens developers

2 Outline Introduction Core Services –VO Management –ACL Management –Shell Service –Discovery –File Access Test framework Portals Performance Project Web Service Description (if time permits)

3 Core development and testing Conrad Steenberg (Python) Michael Thomas (Java) Tahir Azim (Java) Frank van Lingen (Python)

4 Web Services (Framework) A Web Service is a component performing a task, most likely over a network. A Web Service can be identified by a URI and its public interfaces and bindings are described using WSDL. At the basis of a Web Service call (invocation) is a protocol (frequently, but not exclusively this is XML-RPC or SOAP A Web Service Framework is an application that provides support for developing and deploying Web Services.

5 Motivation for Clarens Scientific collaborations are becoming more and more geographically dispersed. –Example CMS Experiment: 2000 Physicists, 150 institutes, 30 countries Web Services identified as an important component to create a scalable globally distributed system Initially Clarens was driven by the CMS community: Provide a framework to support distributed physics analysis

6 Web Service in Clarens Web Service + : –Access Control –Authorization –Discoverable in Distributed System –State management

7 Some of the Functionality needed to Support (Scalable) Distributed Analysis (within CMS) Authentication Access control on Web Services. Remote file access (and access control on files). Discovery of Web Services and Software. Shell service. Shell like access to remote machines (managed by access control lists). Proxy certificate functionality Virtual Organization management and role management. Good performance of the Web Service Framework This is not an exhaustive list List not restricted to CMS but Scientific Collaborations in general

8 dbsvr=Clarens.clarens_client('http://….:80/clarens/')dbsvr.echo.echo('alive?')dbsvr.file.size('index.html')'/web/system','*.html')dbsvr.file.find(['//web'],'*','all') Client Web server Web serverService 3 rd party application Secure cert-based accessed to services through browser http/https Clarens Clarens Clarens XML-RPC SOAP Java RMI JSON RPC

9 Clarens (Python and Java) HTTP Client Tomcat Web Server XML-RPC engine AXIS (SOAP) /xmlrpc servlet JClarens Service Management Remote File Access VO Management Discovery Databases PKI Security Core ServicesUtilities Service Management Remote File Access VO Management Discovery Databases PKI Security Core ServicesUtilities Process Management XML-RPCSOAPGET MOD_PYTHON Apache Web Server HTTP Client (P)Clarens (WAN) Network Configuration

10 Virtual Organization and Role Management Clarens server instance manages tree-like group structure Group administrators authorized to add/delete group members, as well as groups at lower levels Groups can define VOs Subgroup of VOs can define roles. User with multiple roles has DN in multiple groups / Smith 12345 / Entry Examples:


12 Access Control Management Enables administrators to deny or allow groups (VOs) of using resources. ACLs on the server system is controlled by a set of hierarchical ACLs modelled after the access control (.htaccess) files used by Apache –method.service –method.submodule.method In Python Clarens, set ACLs in.clarens_access files

13 Access Control Management (example) ObjectFieldValue modorder allow DNs deny DNs deny groups deny, allow / J / K Physics.LHC.CMS Physics.CDF /O=oldumi/OU=physics/CN=old account Crackers mod.methorder Allow DNs Allow groups Deny DNs Deny groups deny,allow Physics.USA.Caltech Physics.USA.UFL /O=Caltech/OU=CACR/CN=Ed Peng

14 ORDER_ALLOW_DENY=0 ORDER_DENY_ALLOW=1 access=[("",[ORDER_DENY_ALLOW, # Order ["/"], # Allow everybody who can log in [], # Allow group ["/guest"], # Deny indiv default=all [], # Deny default=all [None, None, None]]), ("delete_admin",[ORDER_ALLOW_DENY, # Order [], ["admin"], # Allow group [], # Deny indiv default=all [], # Deny default=all [None, None, None]]), ("list_admin",[ORDER_ALLOW_DENY, # Order [], ["admin"], # Allow group [], # Deny indiv default=all [], # Deny default=all [None, None, None]]), ("auth",[ORDER_ALLOW_DENY, # Order ["/"], [], # Allow group [], # Deny indiv default=all [], # Deny default=all [None, None, None]]) ] # modtime, start_time, end_time


16 Remote File Access Enable scientists access to remote data using well known (file) interfaces Deny or allow read or write access on these remote files, to groups of collaborators. ) ) file.md5( ) file.stat( ) Several File Service Methods


18 Discovery Servers/Services/Software: –Crash –Disappear –Move –Are upgraded –Locally Controlled Dynamic Distributed Environment! register( ) find_server( ) find( ) deregister( ) Discovery Service Interface

19 Discovery MonALISA JINI Network Clarens Servers Clarens Discovery Servers (JINI Clients) Clients CS SS DS Station Servers CL Can be a Clarens Server


21 Shell Access Controlled Access to sites using a shell environment Users DNs are mapped to a local user name Execution of commands/applications in a sandbox File service can be used to navigate sandbox hierarchy

22 Testing Deploy Clarens servers and services, but are they working as intended? Who should be notified when something fails



25 Portal Functionality User's point of access to a Grid system. Provides environment where user can: –Access Grid resources and services. –Execute and monitor Grid applications. –Collaborate with other users. –One stop shop for Grid needs Portals can lower the barrier for users to access Web Services and using Grid enabled applications

26 Clarens Portals Clarens does not have a framework to build portals Portals are dynamic JavaScript based web pages (with authorization and access control) Web server Web serverService 3 rd party application http/https Clarens JavaScript/ HTML GUI JavaScript/ HTML GUI JSON

27 Performance Dual 2.8 GHz Xeon server with 1 GB of memory, accessed 100 Mb/s local area network. Configurable number of unencrypted client connections were opened and set to access the system.list_methods as rapidly as possible. Client: a 2.6 GHz Pentium 4 workstation as a single process opening connections to the server and completing requests asynchronously

28 Performance

29 Projects Using Clarens Ultralight. The network as resource. PEAC. PROOF in a distributed service environment. Physh. Client using web services to access and merge catalog information. SPHINX. Grid scheduler. Lambda Station. Programmatically based access to routers and switches. IGUANA. Graphical Display that can access data through Clarens Web Services. MCPS. Providing Services to submit batch analysis jobs. HotGrid. Gradual access to Grid resources. OSG (Open Science Grid). Uses the Clarens discovery service

30 Third Party Service Example MCPS

31 Provide a simple front end for user to allow them to execute (potentially complex) workflows on the Grid by accessing a Web Service on a tier2. Specify (user exposed parameters) Upload user code and proxy Verify Execute Enable users to use the output of one workflow as input to another (creating their own workflow) Workflows have been specified within Python based on RunJob (Tool developed at FNAL) User should be able to close their laptop and resume later Provide a simple client (first python, in the future a browser interface) to minimize user exposure to Grids and Web Services Design Goals:

32 GenerateSimulateAnalyze Many different generators, and simulations plugin user code Filter 1 (merge) Filter2 (differ) Filter3 (skim) Dataset1 Dataset2 Dataset3 Dataset4 Dataset6 Dataset5 Specified inRunJob and expose several (but not all) parameters to users

33 class Workflow_Example1: def __init__(self): i=0 def execute(self,parameter_values,user_sandbox): # // # // The factory file needs to be pointed at by the SHAHKAR_FACTORY_EXT variable #// f=open(user_sandbox+'/MCPS_test.txt','w') f.write('test succeeded') f.close() # for now we do not have any verification. But if we have they would need # to write the output in a file that can be read by the server. If everything # is ok. this file will be empty. def verify(self,user_parameters,user_sandbox): i=0 How does a workflow specification look like? Specified in a class Two methods: verify and execute Execute contains the meat: The MCPS code and ingests the user parameter choices. Verify can potentially be used to do some MCPS specific verification (optional) Both methods have a user_sandbox parameter which might be needed by MCPS when executing job

34 How does a workflow specification look like? Some real MCPS code in the execute method. def execute(self,parameter_values,user_sandbox): factoryPath = "/home/users/evansde/MCPSInstall/MCPS-dist/Xml/MCPSRunjobFactory.xml" os.environ['SHAHKAR_FACTORY_EXT'] = factoryPath installPath = "/home/users/evansde/MCPSInstall/MCPS-dist/Python" sys.path.append(installPath) from MCPSPython.MCPSUserInterface import createUserInterface mcps = createUserInterface( JobCacheArea = "./jobs", JobName = "Example4-%s" % os.environ['USER'], UniqueID = 1, RuntimePythonBinary = "/usr/bin/python2", ShREEKExecutorOptions = "-v --exitonerror" ) cfgfilesDir = "/home/users/evansde/cfgfiles/" cmkin = mcps.attachExecutable("CMKIN") cmkin['ProjectVersion'] = parameter_values['ProjectVersion'] cmkin['Executable'] = "kine_make_ntpl_pyt6227.exe" cmkin['BeamEnergy'] = 14000. cmkin['NumEvents'] = parameter_values['NumEvents'] cmkin['RunNumber'] = parameter_values['RunNumber'] cmkin['InputCardfile'] = cfgfilesDir + "cmkin-cardfile.txt" cmkin['ExistingInstallation'] = parameter_values['ExistingInstallation'] User parameters (up to the workflow designer to decide what parameters to expose)

35 Future Work Web Start based GUI IM based functionality Integration with dcache (mass storage) and SRM (Storage Resource Broker) Hierarchical access controlled metadata

36 Conclusions Provides a Java/Python based high performance Web Service Framework. Contains a set of core services (file,service management,access control, discovery) needed to support scientific analysis. Adopted by several projects for Web Service development. More information @:

Download ppt "The Clarens Web Service Framework Frank van Lingen (Caltech) on behalf of the Clarens developers."

Similar presentations

Ads by Google