Lofar Survey workshop 8/3/2007 Lofar -Surveys and AstroWise OmegaCEN NOVA – Kapteyn Institute -RuG 8 March 2007 Lofar Survey Workshop OmegaCEN NOVA – Kapteyn.

Slides:

Advertisements

Similar presentations

Microsoft Research Microsoft Research Jim Gray Distinguished Engineer Microsoft Research San Francisco SKYSERVER.

Advertisements

Lehrstuhl Informatik III: Datenbanksysteme Astrometric Matching - E-Science Workflow 1 Lehrstuhl Informatik III: 1 Datenbanksysteme 1 Fakultät für Informatik.

Trying to Use Databases for Science Jim Gray Microsoft Research

1 Online Science The World-Wide Telescope as a Prototype For the New Computational Science Jim Gray Microsoft Research

INAF experience in Grid projects C. Vuerli, G. Taffoni, V. Manna, A. Barisani, F. Pasian INAF – Trieste.

What does LOFAR have to do with the Virtual Observatory (VO)? LOFAR Science Day 16 December 2003 Melbourne David Barnes The University of Melbourne.

Edwin A. Valentijn Prof Astronomical Information Technology Target- OmegaCEN – Kapteyn University of Groningen e-science platform 29 October2013 Target.

ASTRO-WISE Science “A new approach to astronomical archiving & researching for the data- flooded era” Gijs Verdoes Kleijn OmegaCEN, Kapteyn Astronomical.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Infrastructure overview Arnold Meijster &

E-Science Workshop, Santiago de Chile, 23./ KIT ( Frank Schmitz Forschungszentrum Karlsruhe Institut.

NWO Meeting 24 May 04 a Virtual Survey System AstroWise- OmegaCAM – Lofar – Euro-VO a Virtual Survey System AstroWise- OmegaCAM – Lofar – Euro-VO Edwin.

KIDS: mapping the universe with weak lensing Konrad Kuijken, Leiden.

XLDB -Lyon Aug 2009 Astro-WISE -> TarGet OmegaCEN – TarGet Kapteyn Astronomical Institute University of Groningen 28 August 2009 UU OmegaCEN – TarGet Kapteyn.

VO-lecture Valentijn 17/1/2007 Astro-Wise Wide Field imaging Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving,

Digitized Sky Survey Update Brian McLean : Archive Sciences Branch / Operations and Engineering Division.

NORDUnet NORDUnet The Fibre Generation Lars Fischer CTO NORDUnet.

Astro-Wise workshop Nov 2005 Astro-Wise workshop Lorentz center Nov 2005 Munchen, Napoli, Paris, Bonn, Bochum, Nijmegen, Leiden, Groningen Lorentz.

The Japanese Virtual Observatory (JVO) Yuji Shirasaki National Astronomical Observatory of Japan.

VO & Astro-Wise & others A.Belikov OmegaCEN

Functions and Demo of Astrogrid 1.1 China-VO Haijun Tian.

Science Archive for Sky Surveys Data Providers and the VO - NeSC 2003 March Wide Field Astronomy Unit Institute for Astronomy.

Miguel Branco CERN/University of Southampton Enabling provenance on large-scale e-Science applications.

A short introduction to the Worldwide LHC Computing Grid Maarten Litmaath (CERN)

HAH 2007/05/09 Connecting LOFAR to the GRID Hanno Holties, ASTRON IVOA - OGF Astro-RG Work Shop Manchester, 9th May 2007.

Theory in the Virtual Observatory Gerard Lemson, GAVO.

Paul Alexander & Jaap BregmanProcessing challenge SKADS Wide-field workshop SKA Data Flow and Processing – a key SKA design driver Paul Alexander and Jaap.

Astro-WISE & Grid Fokke Dijkstra – Donald Smits Centre for Information Technology Andrey Belikov – OmegaCEN, Kapteyn institute University of Groningen.

10 June 2002Towards an International VO - Garching bei Munchen 1 ASTRO-WISE An Astronomical Wide-field Imaging System for Europe Konrad Kuijken, Edwin.

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks LOFAR Archive Information System Kor Begeman.

INFSO-RI Enabling Grids for E-sciencE Supporting legacy code applications on EGEE VOs by GEMLCA and the P-GRADE portal P. Kacsuk*,

Canadian Virtual Observatory David Schade Canadian Astronomy Data Centre Pat Dowler, Daniel Durand, Luc Simard, Norm Hill, Severin Gaudet.

Astro-WISE workshop 2008 Astro-WISE Astro-WISE workshop Lorentz center Leiden OmegaCEN NOVA – Kapteyn Institute – University Groningen 31 March – 4 April.

Mainframe (Host) - Communications - User Interface - Business Logic - DBMS - Operating System - Storage (DB Files) Terminal (Display/Keyboard) Terminal.

AstroWise((G)A)VO Meeting 6 May 2004 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise.

1 10-June-2004Andy Lawrence : PPARC data curation panel meeting AstroGrid, Data Centres, & Edinburgh What is curation ? Data Centres in the VO era Data.

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Integration of Astro-WISE with Grid storage.

EURO-VO workshop 1 July 2005 A Virtual Survey SYSTEM Astro-Wise NOVA/Kapteyn – OA Capodimonte ESO – Terapix – US Munich/MPE National WFI datacenters NL-I-D-Fr/ESO.

And Tier 3 monitoring Tier 3 Ivan Kadochnikov LIT JINR

EGEE is a project funded by the European Union under contract IST HEP Use Cases for Grid Computing J. A. Templon Undecided (NIKHEF) Grid Tutorial,

Virtual Survey System sept 04 ASTRO-WISE- federation OmegaCEN AstroWise a Virtual Survey System OmegaCAM – Lofar – AstroGrid –((G)A) VO AstroWise a Virtual.

The Virtual Observatory Europe and the VO: the Astrophysical Virtual Observatory and the EURO-VO Astrophysical Virtual Observatory and the EURO-VO Paolo.

GO-ESSP Workshop, LLNL, Livermore, CA, Jun 19-21, 2006, Center for ATmosphere sciences and Earthquake Researches Construction of e-science Environment.

Prologue Major VO objective: “Data at your fingertips” (Lawrence) VO-user’s desire: “going backstage” –Data quality assessment by user (talk Gilmore)

Astro-WISE and pre-calibrated data and external catalogs For the Astro-WISE consortium Gijs Verdoes Kleijn John McFarland.

Solar and space physics datasets within a Virtual Observatory: the AstroGrid experience Silvia Dalla * and Nicholas A Walton  * School of Physics & Astronomy,

…building the next IT revolution From Web to Grid…

E-Science Research Councils awarded e-Science funds ” science increasingly done through distributed global collaborations enabled by the Internet, using.

March 1st, 2006Prospective PNG PNG: Databases - Virtual Observatory.

The Astro-Wise Federation … staying connected Lorentz center, Monday, 31 March 2008.

The International Virtual Observatory Alliance (IVOA) interoperability in action.

German Astrophysical Virtual Observatory Overview and Results So Far W. Voges, G. Lemson, H.-M. Adorf.

AstroGrid NAM 2001 Andy Lawrence Cambridge NAM 2001 Andy Lawrence Cambridge Belfast Cambridge Edinburgh Jodrell Leicester MSSL.

A PPARC funded project Astro-Wise meeting April 2004 OmegaCEN, Kapteyn Institute, Groningen Tony Linde AstroGrid Project Manager University of Leicester.

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarksEGEE-III INFSO-RI Astro-Wise and EGEE.

EURO-VO: GRID and VO Lofar Information System Design OmegaCEN Kapteyn Institute TARGET- Computing Center University Groningen Garching, 10 April 2008 Lofar.

Enabling, facilitating and delivering quality training in the UK and Internationally Introduction to e-science concepts Mike Mineter Training Outreach.

Lofar Information System on GRID A.N.Belikov. Lofar Long Term Archive Prototypes: EGEE Astro-WISE Requirements to data storage Tiers Astro-WISE adaptation.

Introduction to the VO ESAVO ESA/ESAC – Madrid, Spain.

Gijs Verdoes Kleijn Edwin Valentijn Marjolein Cuppen for the Astro-WISE consortium.

NORDUnet NORDUnet e-Infrastrucure: Grids and Hybrid Networks Lars Fischer CTO, NORDUnet Fall 2006 Internet2 Member Meeting, Chicago.

US ATLAS – new grid initiatives John Huth Harvard University US ATLAS Software Meeting: BNL Aug 03.

The AstroGrid-D Information Service Stellaris A central grid component to store, manage and transform metadata - and connect to the VO!

RI EGI-InSPIRE RI Astronomy and Astrophysics Dr. Giuliano Taffoni Dr. Claudio Vuerli.

1 Workflows Access and Massage VO Data José Enrique Ruiz on behalf of the Wf4Ever Team IVOA INTEROP SPRING MEETING 2013 HEIDELBERG, MAY16th 2013.

DutchGrid KNMI KUN Delft Leiden VU ASTRON WCW Utrecht Telin Amsterdam Many organizations in the Netherlands are very active in Grid usage and development,

April 1, 2008Patrick Bos / Kapteyn Astronomical Institute 1 Dwarf galaxies in the ACS Virgo Cluster Survey a bachelor research project by Patrick Bos supervised.

Database Replication and Monitoring

VO – NOVA/NL wp3/wp5 Optical SURVEYS Lofar VST-OMEGACAM:

EGEE NA4 Lofar Lofar Information System Design OmegaCEN

ASTRO-WISE workshop Nov 03

Presentation transcript:

Lofar Survey workshop 8/3/2007 Lofar -Surveys and AstroWise OmegaCEN NOVA – Kapteyn Institute -RuG 8 March 2007 Lofar Survey Workshop OmegaCEN NOVA – Kapteyn Institute -RuG 8 March 2007 Lofar Survey Workshop Edwin A. Valentijn

Lofar Survey workshop 8/3/2007 “Moore’s law” for N-body simulations Courtesy Simon White

Lofar Survey workshop 8/3/2007

Instrument: e-science = share resources beyond workstation Compute power: –AstroWise grid (RuG, Leiden, MPE, Bonn, INAF) –EGEE-3 Storage: – AstroWise fileservers (Leiden, RuG, Nijmegen, MPE, Bonn, INAF) –BiG GRID (RuG- NikHef- SARA) Database: all meta + calibration +source data –AstroWise - Oracle Application programs + doc’s Web services Interoperability: EGEE- EURO-VO = share resources beyond workstation Compute power: –AstroWise grid (RuG, Leiden, MPE, Bonn, INAF) –EGEE-3 Storage: – AstroWise fileservers (Leiden, RuG, Nijmegen, MPE, Bonn, INAF) –BiG GRID (RuG- NikHef- SARA) Database: all meta + calibration +source data –AstroWise - Oracle Application programs + doc’s Web services Interoperability: EGEE- EURO-VO

Lofar Survey workshop 8/3/2007 SDSS ROSAT 2MASS FIRST GAIA Scattered Archives

Lofar Survey workshop 8/3/2007 ROSAT FIRST GAIA SDSS2MASS Euro-VO

Lofar Survey workshop 8/3/2007  Manipulation  X-match  Visualization  Direct links to: CDS Aladin ALADIN

Lofar Survey workshop 8/3/2007 TOPCAT VOTable view

Lofar Survey workshop 8/3/2007 Multi-tier models for LHC & LOFAR Large Hadron Collider Tier-0 (CERN): –copy of all raw data Tier-1 (~8-10 centers) –archive 1/n fraction of raw data & reconstructed data –regular re-processing of the raw data –archiving data from Tier-2 centers –provide central grid services: grid accessible computing and data resources –support coordination Tier-2 (~100 centers): –data analyses –no data archiving Large Hadron Collider Tier-0 (CERN): –copy of all raw data Tier-1 (~8-10 centers) –archive 1/n fraction of raw data & reconstructed data –regular re-processing of the raw data –archiving data from Tier-2 centers –provide central grid services: grid accessible computing and data resources –support coordination Tier-2 (~100 centers): –data analyses –no data archiving LOFAR Tier-0 (RUG): central processor –lots of processing, limited archive Tier-1 (~6-8 centers) –specialized science analysis –advanced processing of (large) data sets –dedicated archive, also for Tier-2 centers –provide central grid services: grid accessible computing and data resources –support coordination Tier-2 (~100 centers): –individual scientists –data analyses –no data archiving

Lofar Survey workshop 8/3/2007 AstroWise paradigm “Classical” paradigm Forward chaining Target processing - Awe Backward chaining waterfall model TIER architecture User hunts upstream driven by input raw data Driven by query of user Process in pipeline workflow Process in bits and pieces on the fly Backward chaining Operators push data User pulls data Results in releases Provide information system Static archives – publish Dynamic archives –publish Internet Raw data - obsolete Raw data is sacred

Lofar Survey workshop 8/3/2007 Virtual Survey Telescope NL:Leiden, Nijmegen, Groningen EU: Napoli, Munchen, Bonn, Heidelberg, Paris Santiago NL:Leiden, Nijmegen, Groningen EU: Napoli, Munchen, Bonn, Heidelberg, Paris Santiago raw pixel data  pipelines/cal files  catalogues all integrated in one information system 100% data lineage 1 dec 2006 delivered ! distributed services processing GRID Storage GRID Methods/services GRID raw pixel data  pipelines/cal files  catalogues all integrated in one information system 100% data lineage 1 dec 2006 delivered ! distributed services processing GRID Storage GRID Methods/services GRID

Lofar Survey workshop 8/3/2007 Current Instruments/projects VST- OmegaCAM –GT, KIDS, VESUVIO, OmegaWhite, OmegaTrans VISTA –UltraVISTA, VIKING via Edinb. Subaru Coma legacy survey Panstarrs Lofar –Surveys –EoR VST- OmegaCAM –GT, KIDS, VESUVIO, OmegaWhite, OmegaTrans VISTA –UltraVISTA, VIKING via Edinb. Subaru Coma legacy survey Panstarrs Lofar –Surveys –EoR

Lofar Survey workshop 8/3/2007 Lofar Storage

Lofar Survey workshop 8/3/2007 Astro-Wise VO Properties Benefits integrated dynamic db on-the fly re-processing 5LS: 5 Lines Script All bits are traced Administration for parallel processing compute GRID Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO on-the fly re-processing 5LS: 5 Lines Script All bits are traced Administration for parallel processing compute GRID Global solutions –astrometry/photometry Build–in workflow Fully user tunable – own provided script Context: projects/surveys, instruments, mydb Publish directly in EURO-VO

Lofar Survey workshop 8/3/2007 TARGET diagram Lofar Survey/ ST ACS

Lofar Survey workshop 8/3/2007

Lofar Survey workshop 8/3/2007

Lofar Survey workshop 8/3/2007 Web services- object viewer

Lofar Survey workshop 8/3/2007 Web services- object maker compute GRID

Lofar Survey workshop 8/3/2007

Planning 2007 ADD with Astron - enable communities Publish WENSS in VO connect to EGEE AstroWise at Astron Build e-SOC at RC RuG Source extraction -> source parameters services Involve Lofar community ADD with Astron - enable communities Publish WENSS in VO connect to EGEE AstroWise at Astron Build e-SOC at RC RuG Source extraction -> source parameters services Involve Lofar community

Lofar Survey workshop 8/3/2007 e-science  me-science