Data mirrors: Techniques and Issues

Slides:

Advertisements

Similar presentations

Trying to Use Databases for Science Jim Gray Microsoft Research

Advertisements

2qz Data Interface Tool Virtual Observatory-India (VO-I) 30/09/2004 2qz Data Interface Tool (DIT)

Harvesting and archiving the Web Nordunet2000, Juha Hakala Helsinki University Library.

Internet Basics. Objectives What is the Internet? What is the Web? How does the Internet work? What do I need to surf the Net? What will I find on the.

1 How To Use a Browser A Module of the CYC Course – Computer Basics

The Australian Virtual Observatory e-Science Meeting School of Physics, March 2003 David Barnes.

Update for CDNL Milan 26 August 2009 Caroline Brazier, Chair of ICADS IFLA-CDNL Alliance for Digital Strategies.

The Internet and the Web

Google and Beyond… Hatch Library Bay Path College / Spring 2010.

Introduction to the Internet

Basic Internet Terms Digital Design. Arpanet The first Internet prototype created in 1965 by the Department of Defense.

What is the Internet? The Internet is a......distributed hypermedia network of networks.

Retrieval of Information from Distributed Databases By Ananth Anandhakrishnan.

Introduction to TCP/IP, the Internet, IP Addressing, and Domain Name.

Development of China-VO ZHAO Yongheng NAOC, Beijing Nov

Introduction to Computing Using Python CSC Winter 2013 Week 8: WWW and Search  World Wide Web  Python Modules for WWW  Web Crawling  Thursday:

Chapter 4 Infrastructure as a Service (IaaS)

Creating web guides for a library portal Jackie Wickham – Intute Martin Gill – University of Leeds

19 Historical overview Main challenge: How to distribute content in high quality over the Internet cost-effectively? • Traditional “Best-effort” model:

Web development  World Wide Web (web) is the Internet system for hypertext linking.  A hypertext document (web page) is an online document. It contains.

Introduction to the Internet How did the Internet start? Why was the Internet developed? How does Internet handle the traffic? Why WWW changed the Internet.

The Savvy Cyber Teacher ® Using the Internet Effectively in the K-12 Classroom 1 Savvy Cyber Administrator ® Using the Internet Effectively in the K-12.

Multimedia & the WWW Week 1 Introduction To….. Today’s Agenda Who I am Who I am Who you are survey & discussion Who you are survey & discussion Course.

What is the Internet? The Internet is a computer network connecting millions of computers all over the world It has no central control - works through.

Leicester Database & Archive Service J. D. Law-Green, J. P. Osborne, R. S. Warwick X-Ray & Observational Astronomy Group, University of Leicester What.

Leicester Database & Archive Service J. D. Law-Green, S. W. Poulton, J. Osborne, R. S. Warwick Dept. of Physics & Astronomy, University of Leicester LEDAS.

DIGITAL Download Sharing and Copying. Digital Download Process of downloading content or materials with the elimination of physical media. (dvd/cdrom)

Introducing the Internet Source: Learning to Use the Internet.

Dr. Omar Al Jadaan The Internet. Internet Service Provider (ISP) Content Providers: create and maintained material that can be accessed using the internet.

Hopkins Storage Systems Lab, Department of Computer Science A Workload-Driven Unit of Cache Replacement for Mid-Tier Database Caching Xiaodan Wang, Tanu.

XHTML Introductory1 Linking and Publishing Basic Web Pages Chapter 3.

Introducción WEB Diseño y programacion en HTML.

Copyright © 2008 Pearson Prentice Hall. All rights reserved. 1 Exploring Microsoft Office Word 2007 Chapter 8 Word and the Internet Robert Grauer, Keith.

1 Session 1: Introduction to HTML Spring Today’s Agenda Cover useful terminology for today’s session HTML, browsers, servers, etc. HTML Tags Get.

CORE 2: Information systems and Databases CENTRALISED AND DISTRIBUTED DATABASES.

THE INTERNET Chapter 13. Internet- Interconnection and Networks “the Net” Computers have played a significant role in our everyday life Growth in popularity.

Chapter 1: The Internet and the WWW CIS 275—Web Application Development for Business I.

Master Thesis Defense Jan Fiedler 04/17/98

Here you are at your computer, but you don’t have internet connections. Your ISP becomes your link to the internet. In order to get access you need to.

An Overview of the Internet: The Internet: Then and Now How the Internet Works Major Features of the Internet.

F. Genova, Berlin 7, Paris, 2 December 2009 The astronomical information network.

NETWORKING and the INTERNET

The SAIC Operation 54 Network and the Internet. Overview The purpose of this brown bag training session is to provide you with an introduction to the.

McLean HIGHER COMPUTER NETWORKING Lesson 15 (a) Disaster Avoidance Description of disaster avoidance: use of anti-virus software use of fault tolerance.

MARCIt records for e-journals project to implement MARCIt service McGill University Library Feb

A National Library for Australian Educational Research Sue Clarke Manager, Cunningham Library Australian Council for Educational Research 27 th IATUL Annual.

April , 2006 HEASARC Users Group Roger Brissenden 1 HEASARC Supported Activities at SAO Roger Brissenden.

WEB BASICS: WHAT IS THE WEB? The Non-Designer’s Design Book by Robin Williams.

Week 2- Overview of the internet The construction of a webpage Four Key Elements – how the internet works Elements and Design concepts Introduction to.

Website Design:. Once you have created a website on your hard drive you need to get it up on to the Web. This is called "uploading“ or “publishing” or.

Session 1: Introduction to HTML Fall Today’s Agenda Talk about the functions of the Internet Cover useful terminology for today’s session HTML,

Web Design Terminology Unit 2 STEM. 1. Accessibility – a web page or site that address the users limitations or disabilities 2. Active server page (ASP)

INTERNET AND . WHAT IS INTERNET The Internet can be defined as the wired or wireless mode of communication through which one can receive, transmit.

INTRODUCTION TO DOCUMENT AUTHORING AND ELECTRONIC PUBLISHING.

Web based Documentation Distribution Tools: MSAccess database (DSN) DreamWeaver Ultradev Microsoft Image Composer Clicking on the document will open an.

Grant Writing for Digital Projects September 2012 IODE Project Office IODE Project Office Oostende, Belgium Oostende, Belgium Sustainability and.

Cofax Scalability Document Version Scaling Cofax in General The scalability of Cofax is directly related to the system software, hardware and network.

(class #2) CLICK TO CONTINUE done by T Batchelor.

Introduction to Information Systems SSD1: Introduction to Information Systems Unit 1. The World Wide Web Unit 2. Introduction to Java and Object- Oriented.

Introduction to the Internet

Evolution of Internet.

ICT Communications Lesson 1: Using the Internet and the World Wide Web

Electronic Resources and the WWW

الوحدة 5 مقدمة في شبكة الانترنت.

Planning Observations

Introduction to the Internet and Web

BMA-IBT-2 Apply technology as a tool to increase productivity to create, edit, and publish industry appropriate documents. 2.3 Execute efficient online.

Unit# 5: Internet and Worldwide Web

The Internet and Electronic mail

Presentation transcript:

Data mirrors: Techniques and Issues By Sarah Ponrathnam sarah@iucaa.ernet.in http://www.iucaa.ernet.in/

Overview Introduction… What is Data Mirroring? Techniques of Mirroring… Motives for Mirroring… Mirror site Issues… Popular Mirroring Tools in Unix… Mirroring Experience at IUCAA…

What is data mirroring? Creating an exact duplicate copy in real-time. In terms of web sites, sites are often mirrored to reduce the traffic on one server.

Techniques of Mirroring Replication is one way to solve availability problem. Distributed Servers Cluster Servers Web site Mirrors

Techniques of Mirroring Distributed Servers: Large web destinations such as google, Yahoo have enough capital to set up and to support distributed servers. Eg. www.google.com, www.google.co.in

Techniques of Mirroring Cluster servers:

Techniques of Mirroring Website Mirrors: Mirrors simply compare and pull the contents from a single master web site at a regular intervals and make identical contents available on another computer, ideally closer to the users making use of it.

Motives for Mirroring Load Balancing - yahoo.co.uk yahoo.co.in High Availability - ADS mirrors Multilingual replication - www.debian.org Database Sharing – www.desert.net www.tucksonweekly.com Franchise / Local Versions - quicken.excite.com quicken.com Virtual Hosting - sports.catalogue.com www.accesports.com

Mirroring Issues Maintenance of mirror-sites in different geographical locations. Integrity of the mirrored contents Providing Host-Independent URLS. Initial transfer of data Optimization Economical constraints Providing Efficient routing

Popular Mirroring tools on Unix RSync http://samba.anu.edu.au/rsync/ Wget http://www.gnu.org/ Mirror http://www.wehlus.de/mirror/download.html

Mirroring Experience IUCAA (Inter University Centre For Astronomy and Astrophysics) works closely with ERNET (Educational and Research NETwork) to make this network a content oriented networkhttp://www.iucaa.ernet.in/

ERNET India ? In 1986, then Department of Electronics had initiated a project "ERNET" with the funding from UNDP. The objective was to establish and operate a nationwide Internet for the Indian academic and research community. Now it has become a full pledged ISP known as ERNET India.

ERNET partners

ERNET Network

ERNET NOC at IUCAA

Mirrors available in IUCAA VizieR provides access to the most complete library of published astronomical catalogues and data tables available on line, organized in a self-documented database. http://urania.iucaa.ernet.in

Mirrors available in IUCAA The NASA Astrophysics Data System (ADS) maintains four bibliographic databases containing more than 4 million records. http://ads.iucaa.ernet.in

VOI Data Archives at IUCAA SDSS Sloan Digital Sky Survey – 1 Tb 2MASS 2 Micron All Sky Survey – 194 Gb 2dfGRS 2 degree field Galaxy Redshift Survey – 5.5 Gb 2QZ 2 Degree field QSO Survey – 630 Mb FIRST Survey Faint Images of Radio Sky at Twenty centimeters – 226 Gb

VOI Hardware

Mirror-site under construction Chandra Data Archive (CDA) 424 GB data is available through ftp service. ftp://cdaftp.iucaa.ernet.in Incremental update is being done regularly. Web based CDA service will be provided, once the new release of CXCDS software is made available.

Acknowledgement Dr. Francois Ochsenbein - VizieR Dr. Guenther Eichhorn and Dr.Alberto Accomazzi - ADS Dr. Ramadurai Padmanabhan - CDA

Questions or Comments?