A Grid Enabled Collaboratory for Scientific Research (GESCR) Charles Severance University of Michigan Sakai Project, NEESGrid Project, OGCE Project

Outline Collaborative Software Historical Context The Sakai Project NEESGrid Experience Comparing Civil Engineering and HENP The Concepts of GESCR Going Forward

Collaboration Happens As individuals, we are parts of many groups and have many roles in those groups

Sakai WLAP OGCE uPortal Architecture Tools Board UM SakaiJune MTG Dec MTG Next Grant Support Database Next Ver JSR-168 FusionGrid NEESGrid Data Model Data Repo Minnesota Texas Version 3Post Oct. HENP GESCR UM Issues One View of Chuck’s Context Map

Another View

Maintaining the Map Read and move to proper folders Copy attachments into folders Searching for information Making calendar entries from

Imagine Software That could create a new “context” in a few clicks –Enroll/invite others to the context as necessary in a few more clicks Context capabilities – list (automatically extracts attachments and places them in folders which appear on your desktop) –Schedule (you can either see a “federated” schedule across all contexts or look at one context) –Persistent browser-based chat - quite useful during meetings when the Polycom or VRVS messes up :) –Resource area where anyone can upload files which appear on everyone’s desktop at the same time (WebDAV) –Threaded discussion area for the context Problem: There are literally hundreds of solutions to portions of this problem.

More Software A single place to see new activity in your “contexts” These contexts are stored on backed-up production servers rather than your desktop for many years A search across your contexts - that would be really cool The ability to customize each context in terms of look, feel, and capabilities The ability to build unique domain specific tools and interfaces to extend the mechanism using Portlets, Servlets, or Applets

A 10-Year Collaborative UM SPARC Science of Collaboratories Sakai Worktools (Notes Based)WTNG Coursetools (Notes Based) CTNGCHEF 1CHEF 2 OGCE Grid Portal NEESGrid

SPARC 2/ users 800 data sources

CourseTools Over 42,000 users at the end of 2003

WorkTools Over 9000 users (2000 active) at the end of 2003

Digital libraries & documents groups-to- information groups-to- facilities people-to-people Communication, Collaboration Services Distributed, media-rich information technology Remote instruments Science of Collaboratories NSF Funded ITR

CHEF 1.0 Fall 2001: CHEF Development begins –Generalized extensible framework for building collaboratories –“Best-of” CourseTools, SPARC, WorkTools Integrate across current UM projects and adopt relevant standards Funded internally at UM as replacement for CourseTools All JAVA - Open Source –Jakarta Jetspeed Portal –Jakarta Tomcat Servlet Container –Jakarta Turbine Service Container Build community of developers through workshops and outreach

Not “just” a portal Portals are a framework to deploy tools (aka rectangles) and focus on how the user wants to arrange their own “rectangles” While CHEF technically is a portal, the goal is for the tools to work together closely and seem to really be parts of a larger “tool” CHEF has a lot of features, (services, presence, notification, etc..) which bridge the gap between portal and application framework

CHEF Applications CourseTools Next Generation WorkTools Next Generation NEESGrid NSF National Middleware Grid Portal

CourseTools Next Generation Over 5000 users at the end of

Worktools Next Generation New WorkTools Sites being created in WTNG as of 12/2003 Run on the same servers as CTNG.

NEESGrid - The Equipment Network for Earthquake Engineering Simulation NSF Funded. NCSA, ANL, USC/ISI, UM, USC, Berkeley, MSU

CHEF-Based NEESGrid Software

NMI / OGCE NSF National Middleware Iniative Indiana, UTexas, ANL, UM, NCSA

What we learned in 10 years. Portal technology is a good idea - forces component approach - functionality does not “smear” Portals are not just aggregators of independent information - but can be an application framework Many (but not all) tools can be used for both teaching and learning and research collaboration Separating functionality into lightweight GUI components and pluggable services with strong and well-specified APIs allows significant reusability GUI elements program to abstract service interfaces - not databases, file systems, LDAP, etc. - this allows great flexibility.

While we were building collaboratories… The Open Knowledge Initiative (OKI) at MIT was developing APIs for learning management systems - involving many universities (UM, Indiana, Stanford, and MIT were strong participants) Indiana, Stanford, MIT all developed learning management system Java Community Process (JCP) produced JSR The “unified” portal standard API Oasis developed the Web Services for Remote Portals (WSRP) standard The open-source uPortal portal project had quietly moved into the #1 open source portal (#4 including commercial vendors)

Jan 04 July 04May 05 Michigan CHEF Framework CourseTools WorkTools Indiana Navigo Assessment Eden Workflow Oncourse MIT Stellar Stanford CourseWork Assessment OKI OSIDs uPortal SAKAI 1.0 Release Tool Portability Profile Framework Services-based Portal Refined OSIDs & implementations SAKAI Tools Complete CMS WorkTools Assessment SAKAI 2.0 Release Tool Portability Profile Framework Services-based Portal SAKAI Tools Complete CMS Assessment Workflow Research Tools Authoring Tools Primary SAKAI Activity Architecting for JSR-168 Portlets, Refactoring “best of” features for tools Conforming tools to Tool Portability Profile Primary SAKAI Activity Refining SAKAI Framework, Tuning and conforming additional tools Intensive community building/training Activity: Ongoing implementation work at local institution… Dec 05 Activity: Maintenance & Transition from a project to a community So we got together and drew an über collaboration picture…

Sakai Core Members Universities –Indiana –Michigan –MIT –Stanford Projects –Open Knowledge Initiative (OKI) –uPortal - JaSIG Funding ($6.8M - 2 Years) –Mellon Foundation –Hewlett Foundation –Partners Program –Core member match KYOU / sakai Boundary, Situation

Sakai Concepts It is neither research nor teaching, it is all “collaboration” - many common tools –Teaching: Courses, tools, drop-boxes –Research: Putting the user interface on the Grid and Virtual Organizations Teaching and Learning Collaborative Research Collaboration and Learning Environment

What we agreed to build… A Collaborative Learning Environment –Open Source –Uses OKI (Open Knowledge APIs) –Uses uPortal as its portal framework Similar to –Blackboard –WebCT And all four core institutions would deploy the commonly developed software

Collaboration and Learning Environment Learning management systems are really just a form of collaboration –Freshman Calculus –Chess Club –Group of 5 faculty members working on curriculum –2000 physics researchers collaborating across the world on a 15-year physics experiment

Sakai 1.0 Site based collaboration environment –Worksite management – Lists –Threaded Discussion –Resources (folders) with WebDav support –Chat –No search yet :( –Many other tools Beta Release July 15, 2004 Production site available at ctools.umich.edu

More Sakai Beta Tools Admin: Alias Editor (chef.aliases) Admin: Archive Tool (chef.archive) Admin: Memory / Cache Tool (chef.memory) Admin: On-Line (chef.presence) Admin: Realms Editor (chef.realms) Admin: Sites Editor (chef.sites) Admin: User Editor (chef.users) Announcements (chef.announcements) Assignments (chef.assignment) C. R. U. D. (sakai.crud) Chat Room (chef.chat) Discussion (chef.discussion) Discussion (chef.threadeddiscussion) Dissertation Checklist (chef.dissertation) Dissertation Upload (chef.dissertation.upload) Drop Box (chef.dropbox) Archive (chef.mailbox) Help (chef.contactSupport) Membership (chef.membership) Message Of The Day (chef.motd) My Profile Editor (chef.singleuser) News (chef.news) Preferences (chef.noti.prefs) Recent Announcements (chef.synoptic.announcement) Recent Chat Messages (chef.synoptic.chat) Recent Discussion Items (chef.synoptic.discussion) Resources (chef.resources) Sample (sakai.module) Schedule (chef.schedule) Site Browser (chef.sitebrowser) Site Info (chef.siteinfo) Web Content (chef.iframe) Worksite Setup (chef.sitesetup) WebDAV

Sakai Going Forward Focus on the”Learning” of Collaborative Learning Environment through 2Q05 –Getting ready for production deployment at the four partner sites –Improving the look and feel of the software –Many feature enhancements (to satisfy four + 60 schools) –New GUI Programming Environment based on Java Server Faces –Building new set of Sakai APIs (Java) Based on OKI - Enabling RDF Move into OGCE and NEESGrid starting 3Q04 Release Q04

The George E. Brown, Jr. Network for Earthquake Engineering Simulation (NEES)

NEES Founding George E. Brown, Jr. Network for Earthquake Engineering Simulation (NEES). Funded in > $100M Goal: Transform the nation’s ability to carry out earthquake engineering research, to obtain information vital to develop improved methods for reducing the nation’s vulnerability to catastrophic earthquakes, and to educate new generations of engineers, scientists and other specialists committed to improving seismic safety. To be Completed: October 2004

NEESgrid facilitates research capabilities previously unavailable NEESgrid links earthquake researchers across the U.S. with leading-edge computing resources and research equipment and allowing collaborative teams (including remote participants) to plan, perform, and publish their experiments NEESgrid is a coordinated and secure architecture/environment NEESgrid is a modular and extensible environment with a customizable user interface NEESgrid provides common tools that allow leveraging resources and experiences Rather than having to worry about the required cyber infrastructure, NEESgrid allows researchers to focus on the earthquake engineering challenges at hand The goal of the System Integrator (SI) is to develop NEESgrid as the Cyber Infrastructure that will facilitate this next generation of experimentation/simulation in earthquake engineering

NEES Components New experimental facilities (15) –Oregon State University, Rensselaer Polytechnic Institute, University of Buffalo, University of Colorado at Boulder, University of Minnesota, University of Nevada at Reno, University of Texas at Austin, and the University of California campuses at Berkeley, Davis and Los Angeles Collaborative Software System: NEESGrid –Collaboration –Data capture and sharing –Tele-presense and Tele-operation –Simulation –Support for Hybrid Simulation and Physical Experiments

Shake table: Nevada, Reno

Reaction wall: Minnesota

Centrifuge: UC Davis

Wave basin: Oregon State

Field structural: UCLA

Field geotechnical: Texas

If we build it, they will collaborate Data and access to data represent fundamental barriers to dispersed collaboration Efficient movement of vast amounts of data is a prime rationale for cyberinfrastructure Federating, visualizing and mining data are principle challenges

Researchers DataFacilities n Synchronized data n Synchronized data and images n Data discovery n Automatic archiving n Simulation codes n Hybrid experiments n Teleoperation n Teleobservation n Synchronous communication n Asynchronous communication The collaboratory concept

NEES Resources Field Equipment Laboratory Equipment Remote Users Remote Users: ( K-12 Faculty and Students) Instrumented Structures and Sites Leading Edge Computation Curated Data Repository Laboratory Equipment Global Connections ( Faculty, Students, Practitioners) Simulation Tools Repository

The Grid in NEESgrid NEESpop A Experimental Equipment Video I/O Audio I/O Site A: Experimental Data Producer Hub A Telepresence Equipment Active PI Data Cache Site B: Remote Lead Investigator Hub B Internet Fabric and Operations Site C: Passive Collaborator Teleobservation Equipment Passive co-PI Data Cache Hub C Grid Data Repository Grid Operations Center Experimental Component Campus Net Component NEESgrid Component

NEESGrid Software Founding NMI Technologiess –Globus Toolkit –OGCE Collaboration Toolkit New Work –Data and Metadata Repository - NCSA –Data Acquisition, Storage, and Visualization –Simulation Portal –Synchronized data and video (live and stored)

NEESgrid Experiment Data Flow NEESGrid Data Repository Project Browser Data Turbine Data Ingestion Experiment Control Streaming Viewer DAQ CD SiteSpecific ProjectRelated ExperimentalSetup ExperimentalElement DataElement Data Model DAQ Disk Stored Viewer

DT Main System PTZ/ USB Still Capture DT Client BT848 Video Frames DT Client Capturing Video and Data Camera Control Gateway DAQ Data Capture DT Client Simulation Coordinator Site A Site B

DT Main System Data Monitoring Tools Still Image / Camera Control ~ <> ^ ^ <> Camera Control Gateway Creare viewers Still image camera control Thumb- nail

Video and Data Tivo Thumbnail + Audio + Data <>+

Simulation Coordinator NTCP Server DAQ Comp Sim DAQ NTCP Server Live Extractor Quicktime DAQ Data Data Repository Sample Experiment Setup

Data Models Data models are developed in RDF Local repository supports multiple simultaneous data models with cross-model linkages Metadata browser (aka Project browser) becomes the Project Browser, Notebook Browser, Site Specification Database Browser Metadata browser can federate multiple sources of Metadata

Overall Data Modeling Efforts NEES Site ASite CSite B Equipment People Experiments Trials EquipmentPeople ExperimentsTrials Data Tsnumai Specimen Shake Table Specimen Geotech Specimen Centrifuge Specimen UnitsSensors Descriptions Site Specifications Database Project Description Domain Specific models Common Elements Data / Observations Ref. Source: Chuck Severance

Models + Data Model Repo Models Configure Data Load Configure RDF/ OWL RDF

Protégé - 2K Models + Data Model Repo Models Configure Data Load Configure RDF/ OWL RDF

DOE ELN / Example

NEESgrid: Simulation

Lessons Learned in NEESGrid (Chuck’s views) A collaborative environment *must* have a user interface Just asking about requirements is a tiny part of the problem - CS developers must learn to “walk in the shoes” of the scientists. Scientists know a lot about Computer Science - listen to them and involve them The real work begins after software is “delivered” - Version 1.0 is usually just a conversation starter - but a very important step There are some things that are useful across fields - but the most valuable elements are field-unique Data models, data repositories, and long-term curation is difficult!

HENP/Civil Similarities

HENP/Civil: Differences Willingness to collaborate –CE: Not too keen on the idea - NSF forcing the issue using “carrot and stick” –HENP: It is part of the fabric of the field (at least within Atlas and CMS :) ) Technology Savvy –CE: It is all about the “within lab” electronics –HENP: In lab electronics is difficult, important and different. The internet is a tool to be used both for human communication and data manipulation - have tried everything - use simple reliable stuff because the work cannot wait.

NEES Going Forward Release 3.0 complete July 31, 2004 –Improve documentation and bug fixes through September 30, 2005 As of October 1, 2004, NEESGrid will transition from development lead by NCSA to maintenance led by SDSC –Focus on deployment and support –UM will continue involvement - move to Sakai - improving data modeling, internationalization (Japan)

GESCR* - The Vision * A Grid Enabled Collaboratory for Scientific Research

HENP: Unique Aspects Already highly collaborative as a scientific field Globally distributed in space and time (both time-zones and years) Technology savvy but wary :) Solutions underway (iVDGL, GriPhyn, etc) without significant user interface component

GESCR in a sentence.. ….. will combine the best open-source applications from within the HENP communities (and from allied Grid and National Middleware efforts) within a common portal interface. Put another way: There is a lot of work going on in HENP that does not really have much of a user interface yet. We want to put a user interface on the Grid and other activities.

GESCR in a sentence.. ….. will combine the best open-source applications from within the HENP communities (and from allied Grid and National Middleware efforts) within a common portal interface. What I tell people: GESCR will be a tri-corder for the Atlas and CMS collaborations. Starting with inter-human communication and moving towards data processing and analysis.

Elements of GESCR Sakai with Grid Support (from OGCE / NEESGrid) Sakai Collaboration Tools Well funded analysis of application, requirements, and challenges throughout the project Additional new capabilities –VRVS Integration (rooms, record, playback) –Web Lecture Archive (WLAP) –VNC support

GESCR Elements (cont) MonaLisa integration Collaborative LaTeX editing environment Peer-to-Peer file sharing Language of Access to handle “over- collaboration” Adaption of HEPBook GESCR support office (Y3) Education and outreach activities

GESCR Project Planned: Four years - Medium ITR (NSF) Partners: Caltech, Fermilab, Florida International University, Lawrence Berkeley, Maryland, University of Michigan, University of Texas Austin, University of Iowa Joint between Atlas and CMS Has been submitted twice to the NSF ITR program at the urging of the Physics directorate to create a CMS/Atlas joint IT effort

GESCR - Going Forward Waiting for funding decision on most recent submission –If funded, work on unique physics capabilities (VRVS, MonaLISA, etc) will begin immediately –If not funded, leverage NEESGrid / Open Grid Computing Environment / Sakai - focus on the generic tools - deploy in a physics context Immediately, we can begin some experimental usage for interested groups using the University of Michigan CTools service

University of Michigan: Ctools University of Michigan runs a large scale professional server cluster for teaching and learning and research collaboration –Research collaboration simply needs a single UMich member of each group - the rest of the members have no- charge “friend” accounts based on addresses. –For multiple accounts, it is best to have a UM person to act as your “problem solver” familiar with your group. If there is interest, we can provide staff from the MGrid ( organization to support HEP collaborative activities. ctools.umich.edu

Sakai-Based UM In production August 1, 2004.

Summary We will deploy collaboration software for Atlas / CMS sooner or later If / when we get funding it will accelerate the process and provide Physics specific capabilities Collaborative portals is a well-funded and well-coordinated activity (Sakai 30+FTE, NEESGrid 10+FTE, OGCE 5+ FTE) It is (almost) a no-lose situation - the longer we wait, the more mature the software will be

Questions Thank you for your time… Questions to