High Throughput Data Program (HTDP) at FNAL Mission: investigate the impact of and provide solutions for the scientific computing challenges in Big Data.

Slides:

Advertisements

Similar presentations

HOlistic Platform Design for Smart Buildings

Advertisements

IMPLEMENTING FINANCIAL AND ACCOUNTING SYSTEMS FOR GOVERNMENT CHRISTIAN T. SOTTIE THE CONTROLLER AND ACCOUNTANT-GENERAL GHANA.

Achieve Benefit from IT Projects. Aim This presentation is prepared to support and give a general overview of the ‘How to Achieve Benefits from IT Projects’

High Throughput Data Program at Fermilab R&D Parag Mhashilkar Grid and Cloud Computing Department Computing Sector, Fermilab Network Planning for ESnet/Internet2/OSG.

Current Testbed : 100 GE 2 sites (NERSC, ANL) with 3 nodes each. Each node with 4 x 10 GE NICs Measure various overheads from protocols and file sizes.

 Contributing >30% of throughput to ATLAS and CMS in Worldwide LHC Computing Grid  Reliant on production and advanced networking from ESNET, LHCNET and.

MIcro-NanOSystems ECAD Laboratory GOSPEL : General Olfaction and Sensing Projects on a European Level Projects on a European Level A Network.

Integrating Network and Transfer Metrics to Optimize Transfer Efficiency and Experiment Workflows Shawn McKee, Marian Babik for the WLCG Network and Transfer.

The LHC Computing Grid Project Tomi Kauppi Timo Larjo.

Capacity building for public authorities – EE7 Veronika Czako Project Adviser, EASME.

H-1 Network Management Network management is the process of controlling a complex data network to maximize its efficiency and productivity The overall.

Thomas Hacker Barb Fossum Matthew Lawrence Open Science Grid May 19, 2011.

Assessment of Core Services provided to USLHC by OSG.

F Run II Experiments and the Grid Amber Boehnlein Fermilab September 16, 2005.

Catherine Bowen, Policy & Stakeholder Director. What is the NBCS? The National Business Crime Solution (NBCS) is a ‘Not for Profit’ Initiative that provides.

October 24, 2000Milestones, Funding of USCMS S&C Matthias Kasemann1 US CMS Software and Computing Milestones and Funding Profiles Matthias Kasemann Fermilab.

1 Robert S. Webb and Roger S. Pulwarty NOAA Climate Service.

EGI-Engage EGI-Engage Engaging the EGI Community towards an Open Science Commons Project Overview 9/14/2015 EGI-Engage: a project.

Andrea Ricci - ISIS Brussels, 12 April 2012 Smart Grids: Overview of the study and main challenges 1.

10/20/05 LIGO Scientific Collaboration 1 LIGO Data Grid: Making it Go Scott Koranda University of Wisconsin-Milwaukee.

HIM Breaking Into Informatics Mari Pirie-St. Pierre, MS, RHIA.

Long Term Ecological Research Network Information System LTER Grid Pilot Study LTER Information Manager’s Meeting Montreal, Canada 4-7 August 2005 Mark.

material assembled from the web pages at

100G R&D at Fermilab Gabriele Garzoglio (for the High Throughput Data Program team) Grid and Cloud Computing Department Computing Sector, Fermilab Overview.

May 8, 20071/15 VO Services Project – Status Report Gabriele Garzoglio VO Services Project – Status Report Overview and Plans May 8, 2007 Computing Division,

100G R&D at Fermilab Gabriele Garzoglio Grid and Cloud Computing Department Computing Sector, Fermilab Overview Fermilab Network R&D 100G Infrastructure.

100G R&D at Fermilab Gabriele Garzoglio Grid and Cloud Computing Department Computing Sector, Fermilab Overview Fermilab Network R&D 100G Infrastructure.

Page 1Prepared by Sapient for MITVersion 0.1 – August – September 2004 This document represents a snapshot of an evolving set of documents. For information.

10/24/2015OSG at CANS1 Open Science Grid Ruth Pordes Fermilab

D0RACE: Testbed Session Lee Lueking D0 Remote Analysis Workshop February 12, 2002.

Interoperability Grids, Clouds and Collaboratories Ruth Pordes Executive Director Open Science Grid, Fermilab.

The Earth System Grid (ESG) Computer Science and Technologies DOE SciDAC ESG Project Review Argonne National Laboratory, Illinois May 8-9, 2003.

1 European e-Infrastructure experiences gained and way ahead OGF 20 / EGEE User’s Forum 9 th May 2007 Mário Campolargo European Commission - DG INFSO Head.

Spectrum of Support for Data Movement and Analysis in Big Data Science Network Management and Control E-Center & ESCPS Network Management and Control E-Center.

ICT TOOLS AND SOCIETY INVOLVEMENT AMONG THE EUPAN NETWORK HIGHLIGHTS FROM THE SURVEY RESULTS TANYA CHETCUTI AND MARCO FICHERA - WORKSHOP EUROPEAN COMMISSION.

CERN IT Department CH-1211 Geneva 23 Switzerland t GDB CERN, 4 th March 2008 James Casey A Strategy for WLCG Monitoring.

TechCon Food systems history… Agriculture has a 10,000 year history Farmers are estimated to be 38 to 45% of the global work force In the developing.

Ruth Pordes November 2004TeraGrid GIG Site Review1 TeraGrid and Open Science Grid Ruth Pordes, Fermilab representing the Open Science.

EGEE-III INFSO-RI Enabling Grids for E-sciencE Ricardo Rocha CERN (IT/GS) EGEE’08, September 2008, Istanbul, TURKEY Experiment.

Open Science Grid & its Security Technical Group ESCC22 Jul 2004 Bob Cowles

High Energy FermiLab Two physics detectors (5 stories tall each) to understand smallest scale of matter Each experiment has ~500 people doing.

Introducing the Open Science Grid Project supported by the Department of Energy Office of Science SciDAC-2 program from the High Energy Physics, Nuclear.

Sep 25, 20071/5 Grid Services Activities on Security Gabriele Garzoglio Grid Services Activities on Security Gabriele Garzoglio Computing Division, Fermilab.

Parag Mhashilkar Computing Division, Fermi National Accelerator Laboratory.

The Claromentis Digital Workplace An Introduction

FROM PRINCIPLE TO PRACTICE: Implementing the Principles for Digital Development Perspectives and Recommendations from the Practitioner Community.

8a Certified. About Us  Headquarters in Vienna, VA  Service Disabled Veteran-owned Small Business  SBA 8(a) program participant  Small Disadvantaged.

Open Science (publishing) as-a-Service Paolo Manghi (OpenAIRE infrastructure) Institute of Information Science and Technologies Italian Research Council.

1 Open Science Grid: Project Statement & Vision Transform compute and data intensive science through a cross- domain self-managed national distributed.

100G R&D for Big Data at Fermilab Gabriele Garzoglio Grid and Cloud Computing Department Computing Sector, Fermilab ISGC – March 22, 2013 Overview Fermilab.

Monitoring and Information Services Core Infrastructure (MIS-CI) Service Description Mark L. Green OSG Integration Workshop at UC Feb 15-17, 2005.

ADVANCED TRANSPORTATION AND CONGESTION MANAGEMENT TECHNOLOGIES DEPLOYMENT (ATCMTD) PROGRAM 1 Bob Arnold, Director Office of Transportation Management,

Building on virtualization capabilities for ExTENCI Carol Song and Preston Smith Rosen Center for Advanced Computing Purdue University ExTENCI Kickoff.

Parag Mhashilkar Computing Division, Fermilab.  Status  Effort Spent  Operations & Support  Phase II: Reasons for Closing the Project  Phase II:

The opportunities and challenges of sharing genomics data with the pharmaceutical industry Shahid Hanif, Head of Health Data & Outcomes, ABPI DNA digest.

100G R&D at Fermilab Gabriele Garzoglio (for the High Throughput Data Program team) Grid and Cloud Computing Department Computing Sector, Fermilab Overview.

Big Data over a 100G Network at Fermilab Gabriele Garzoglio Grid and Cloud Services Department Computing Sector, Fermilab CHEP 2013 – Oct 15, 2013 Overview.

Scientific Computing at Fermilab Lothar Bauerdick, Deputy Head Scientific Computing Division 1 of 7 10k slot tape robots.

Issues in Cloud Computing. Agenda Issues in Inter-cloud, environments  QoS, Monitoirng Load balancing  Dynamic configuration  Resource optimization.

Eric Peirano, Ph.D., TECHNOFI, COO

Gene Oleynik, Head of Data Storage and Caching,

Eric Peirano, Ph.D., TECHNOFI, COO

OVERVIEW What is Ashoka? What is Changemakers.com?

Scaling Science Communities Lessons learned by and future plans of the Open Science Grid Frank Würthwein OSG Executive Director Professor of Physics UCSD/SDSC.

Open Science Grid Progress and Status

EOSC MODEL Pasquale Pagano CNR - ISTI

Providing Big Data to facilitate Collaboration in a Safety Railway Environment Jens-Peter Brauner, VP Mobility Division, Siemens Ltd. Hong Kong 27th International.

Grid Portal Services IeSE (the Integrated e-Science Environment)

New Relic Digital Intelligence Platform 1 Operational Efficiency with Full Stack Visibility Monitor the real-time impact of your IT ecosystem.

What is collaboration? Turf Trust Compet e Co-exist Communicate

Presentation transcript:

High Throughput Data Program (HTDP) at FNAL Mission: investigate the impact of and provide solutions for the scientific computing challenges in Big Data for the the Computing Sector at Fermilab and its stakeholders. Three major thrusts driving the project 1. Grid middleware evaluation on 100GE to prepare lab and its stakeholders for the transition (Stakeholders: US-CMS, IF, DES) 2. Evolve the Network towards a manageable resource. Participate in OSG Networking activity (Stakeholders: OSG, USCMS) 3. Integration of network intelligence with data and job management middleware (Stakeholders: USCMS) 1

The GCC dept. participates in the OSG networking area with the vision of Network-As-A-Service  OSG Networking Dashboard – a repository of network information  Secure Information gathering from a PerfSONAR network  Integration layer  Secure Presentation layer: Information consumption by human and middleware  We are ramping up to contribute to the development of the dashboard (internal stakeholder: USCMS)  Participation in SubGroups (Gabriele organizer of the User AuthN / AuthZ; Parag of User API and Use Cases).  Plan to add GridFTP network data to the repository.  Dashboard as source of information for network intelligence 2 2. Participation in the OSG Networking Activity

1. Goals of 100 GE 3 End-to-end experiment analysis systems include a deep stack of software layers and services. Need to ensure these are functional and effective at the 100 GE scale.  Determine and tune the configuration to ensure full throughput in and across each layer/service.  Measure and determine efficiency of the end-to-end solutions.  Monitor, identify and mitigate error conditions.

3. Integration of network intelligence with data management service Feed network knowledge to the decisions on data and job placement (e.g. think of replica selection) Consumes information from the Dashboard Present information in a format that can be easily consumed by data management middleware Seeking collaborators to complement funding from external sources (targeting ASCR next) 4