Presentation is loading. Please wait.

Presentation is loading. Please wait.

INFSO-RI-508833 Enabling Grids for E-sciencE SRMv2.2 in DPM Sophie Lemaitre Jean-Philippe.

Published byOpal Byrd Modified over 8 years ago

Similar presentations

Presentation on theme: "INFSO-RI-508833 Enabling Grids for E-sciencE SRMv2.2 in DPM Sophie Lemaitre Jean-Philippe."— Presentation transcript:

1 INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org SRMv2.2 in DPM Sophie Lemaitre (Sophie.Lemaitre@cern.ch)Sophie.Lemaitre@cern.ch Jean-Philippe Baud (Jean-Philippe.Baud@cern.ch)Jean-Philippe.Baud@cern.ch SRMv2.2 Workshop Edinburgh – 13th November 2007

2 Enabling Grids for E-sciencE INFSO-RI-508833 2 My message SRMv2.2 –No migration needed! –Just make sure the SRMv2.2 daemon is running Space tokens vs. User space token descriptions SRMv2.2 useful tools –dpm-reservespace –dpm-updatespace –dpm-releasespace

3 Enabling Grids for E-sciencE INFSO-RI-508833 3 Outline DPM architecture Authorization –In the Namespace –On pools SRMv2.2 in DPM Log files Admin corner

4 Enabling Grids for E-sciencE INFSO-RI-508833 4 Outline DPM architecture Authorization –In the Namespace –On pools SRMv2.2 in DPM Log files Admin Corner

5 Enabling Grids for E-sciencE INFSO-RI-508833 5 DPM architecture DPM CLI, C API, SRM-enabled client, etc. disk01.cern.ch:/data01/dteam/2007-06-03/my_file.150.0 /dpm/cern.ch/home/dteam/my_file

6 Enabling Grids for E-sciencE INFSO-RI-508833 6 DPM architecture (details) CLI, C API, SRM-enabled client, etc. data transfer DPM head node /domain /vo file SRMv1SRMv2SRMv2.2 DPMDPNS /dpm /home DPM disk servers … Secure RFIO GridFTP Secure RFIO GridFTP DPM database DPNS database

7 Enabling Grids for E-sciencE INFSO-RI-508833 7 Ports –DPNS:5010 –DPM:5015 –SRMv1:8443 –SRMv2:8444 –SRMv2.2:8446

8 Enabling Grids for E-sciencE INFSO-RI-508833 8 DPM Databases Name Server database –Namespace  Cns_file_metadata –Authorization  Cns_groupinfo, Cns_userinfo –Replicas  Cns_file_replica DPM database –DPM configuration  dpm_pool, dpm_fs –All requests (SRM, transfers, etc.)  dpm_pending_req, dpm_req, dpm_get_filereq, dpm_put_filereq –Space reservation  dpm_space_reserv

9 Enabling Grids for E-sciencE INFSO-RI-508833 9 Outline DPM architecture Authorization –In the Namespace –On pools Log files SRMv2.2 in DPM Admin corner

10 Enabling Grids for E-sciencE INFSO-RI-508833 10 DPM virtual ids Each user’s DN –Is mapped to a unique virtual uid Each VOMS group, each VOMS role –Is mapped to a unique virtual gid Virtual uids / gids are created automatically –the first time a given user / group contacts the DPM DPM Name Server (uid1, gid1)

11 Enabling Grids for E-sciencE INFSO-RI-508833 11 DPM virtual ids /C=CH/O=CERN/OU=GRID/CN=Sophie Lemaitre 2268101 /C=CH/O=CERN/OU=GRID/CN=Simone Campana 7461102 Virtual gids mapping (example) Virtual uids mapping (example) atlas101 atlas/Role=lcgadmin102 atlas/Role=production103 DPM Name Server (uid1, gid1) Ex: (102, 101) $ grid-proxy-init $ voms-proxy-init --vo atlas  Simone will be mapped to (uid, gid) = (102, 101) DB

12 Enabling Grids for E-sciencE INFSO-RI-508833 12 DPM secondary groups /C=CH/O=CERN/OU=GRID/CN=Sophie Lemaitre 2268101 /C=CH/O=CERN/OU=GRID/CN=Simone Campana 7461102 Virtual gids mapping (example) Virtual uids mapping (example) atlas101 atlas/Role=lcgadmin102 atlas/Role=production103 DPM Name Server (uid1, gid1) Ex: (102, 103, 101) $ voms-proxy-init –voms atlas:/atlas/Role=production  Simone will be mapped to (uid, gid, …) = (102, 103, 101)  Simone still belongs to “atlas” DB

13 Enabling Grids for E-sciencE INFSO-RI-508833 13 Outline DPM architecture Authorization –In the Namespace –On pools SRMv2.2 in DPM Log files Admin corner

14 Enabling Grids for E-sciencE INFSO-RI-508833 14 DPM pools DPM terminology –A DPM pool is a set of dedicated filesystems on DPM disk servers By default, pools are generic Possibility to dedicate a pool to one or several groups –dpm-addpool --poolname poolA --group alice –dpm-addpool --poolname poolB --group atlas,cms,lhcb Easy to add or remove groups –dpm-modifypool --poolname poolA --group +atlas,-alice Primary group only for writing –For disk space accounting –Who actually uses the space gets the bill…

15 Enabling Grids for E-sciencE INFSO-RI-508833 15 Outline DPM architecture Authorization –In the Namespace –On pools SRMv2.2 in DPM Log files Admin corner

16 Enabling Grids for E-sciencE INFSO-RI-508833 16 Migration to SRMv2.2? No “migration to SRMv2.2” needed –No need to update the DPM namespace  All files are accessible via SRMv1, v2 and v2.2 Same logical file names Same physical files  The SRM daemons act as front end interfaces, they don’t affect the namespace SRMv2.2 daemon needs to run –Automatically done by YAIM –You are probably already running SRMv2.2!

17 Enabling Grids for E-sciencE INFSO-RI-508833 17 File Storage Types –Volatile –Permanent –Any (default) Some client tools use “volatile” type, others “permanent” –Best to configure your pools as “-” (default)  dpm-addpool --poolname MyPool --def_filesize 200M --s_type “-”

18 Enabling Grids for E-sciencE INFSO-RI-508833 18 Retention Policy Retention policies –Given quality of disks, admin defines quality of service –Replica (default), Output, Custodial Example –dpm-modifypool --poolname MyPool --ret_policy R

19 Enabling Grids for E-sciencE INFSO-RI-508833 19 Space concepts Space token –A UUID created internally by the DPM  Ex: fe869590-b771-4002-b11a-8e7430d72911 User space token description –A case sensitive description –Provided by a user or the DPM admin  Ex: myspace, atlas_prod, TutorialSpace No relationship between –spaces, and –the DPM namespace

20 Enabling Grids for E-sciencE INFSO-RI-508833 20 Space Reservation Static space reservation (admin) –No limitation on duration and size of space reservation $ dpm-reservespace --gspace 20G --lifetime Inf --group atlas --token_desc Atlas_ESD $ dpm-reservespace --gspace 100M --lifetime 1h --group dteam/Role=lcgadmin --token_desc LcgAd $ dpm-updatespace --token_desc myspace --gspace 5G $ dpm-releasespace --token_desc myspace Dynamic space reservation (user) –Defined by user on request  dpm-reservespace, dpm-updatespace, dpm-releasespace  srmReserveSpace –Limitation on duration and size of space reservation Features released since DPM 1.6.3

21 Enabling Grids for E-sciencE INFSO-RI-508833 21 Outline DPM architecture Authorization –In the Namespace –On pools SRMv2.2 in DPM Log files Admin corner

22 Enabling Grids for E-sciencE INFSO-RI-508833 22 DPM log files Example: DPNS log  /var/log/dpns/log 11/25 11:19:55 2050,0 Cns_srv_listreplica: NS092 - listreplica request by /C=IT/O=INFN/OU=Personal Certificate/L=Bari/CN=Nicola De Filippis/Email=Nicola.defilippis@ba.infn.it (42043,2690) from t2-srm-01.lnl.infn.it 11/25 11:19:55 2050,0 Cns_srv_listreplica: NS098 - listreplica /dpm/lnl.infn.it/home/cms/store/unmerged/mc/2006/11/9/mc-physval-111- SingleMuPlus-Pt5To200/GEN-SIM-DIGI-RECO/30000/BC8936E9-C678-DB11- AFE6-00096BB5CC34.root 11/25 11:19:55 2050,3 Cns_srv_setratime: NS092 - setratime request by /C=IT/O=INFN/OU=Personal Certificate/L=Bari/CN=Nicola De Filippis/Email=Nicola.defilippis@ba.infn.it (42043,2690) from t2-srm-01.lnl.infn.it 11/25 11:19:55 2050,0 Cns_srv_listreplica: returns 0 Log Example Virtual uid, gid Thread #3 Thread #0

23 Enabling Grids for E-sciencE INFSO-RI-508833 23 SRMv2.2 log file 11/05 05:06:16 25696,0 PrepareToPut: request by /C=IT/O=INFN/OU=Personal Certificate/L=Pisa/CN=Flavia Donno/Email=flavia.donno@pi.infn.it from lxdev25.cern.ch 11/05 05:06:16 25696,0 PrepareToPut: SRM98 - PrepareToPut 354486 b2c7ba01-ca72-4c9a-9714-312e70db7f22 11/05 05:06:16 25696,0 PrepareToPut: SRM98 - PrepareToPut 0 srm://lxdpm102.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/20071105-050615-1865/f0 11/05 05:06:16 25696,0 PrepareToPut: SRM98 - PrepareToPut 1 srm://lxdpm102.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/20071105-050615-1865/f1 11/05 05:06:16 25696,0 PrepareToPut: returns 0, statusCode=SRM_REQUEST_QUEUED 11/05 05:06:16 25696,0 StatusOfPutRequest: request by /C=IT/O=INFN/OU=Personal Certificate/L=Pisa/CN=Flavia Donno/Email=flavia.donno@pi.infn.it from lxdev25.cern.ch 11/05 05:06:16 25696,0 StatusOfPutRequest: SRM98 - StatusOfPutRequest b2c7ba01-ca72-4c9a-9714-312e70db7f22 11/05 05:06:16 25696,0 StatusOfPutRequest: SRM98 - StatusOfPutRequest srm://lxdpm102.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/20071105-050615-1865/f0 11/05 05:06:16 25696,0 StatusOfPutRequest: SRM98 - StatusOfPutRequest srm://lxdpm102.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/20071105-050615-1865/f1 11/05 05:06:16 25696,0 StatusOfPutRequest: returns 0, statusCode=SRM_REQUEST_INPROGRESS 11/05 05:09:17 25696,0 StatusOfPutRequest: request by /C=IT/O=INFN/OU=Personal Certificate/L=Pisa/CN=Flavia Donno/Email=flavia.donno@pi.infn.it from lxdev25.cern.ch 11/05 05:09:17 25696,0 StatusOfPutRequest: SRM98 - StatusOfPutRequest b2c7ba01-ca72-4c9a-9714-312e70db7f22 11/05 05:09:17 25696,0 StatusOfPutRequest: SRM98 - StatusOfPutRequest srm://lxdpm102.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/20071105-050615-1865/f0 11/05 05:09:17 25696,0 StatusOfPutRequest: SRM98 - StatusOfPutRequest srm://lxdpm102.cern.ch:8446/srm/managerv2?SFN=/dpm/cern.ch/home/dteam/20071105-050615-1865/f1 11/05 05:09:17 25696,0 StatusOfPutRequest: returns 0, statusCode=SRM_SUCCESS /var/log/srmv2.2/log example SRMv2.2 status codes SRMv2.2 calls received

24 Enabling Grids for E-sciencE INFSO-RI-508833 24 Admin corner Useful tools –DPM check consistency tool –DPM request database cleanup tool No need to restart the daemons when the host certificate changes SLC3 to SLC4 migration –OS reinstallation, not an update –Backup the MySQL “cns_db” and “dpm_db” databases –Need to backup:  Log files  Configuration files (NSCONFIG, DPMCONFIG)  etc. –Follow this procedure:  https://twiki.cern.ch/twiki/bin/view/LCG/DpmMigratingFromSL3TOSL4 https://twiki.cern.ch/twiki/bin/view/LCG/DpmMigratingFromSL3TOSL4

25 Enabling Grids for E-sciencE INFSO-RI-508833 25 TO DO list On the TO DO list –Blacklisting via LCAS –ChangeSpaceForFiles  To be able to move a file from one space to another –Quotas

26 Enabling Grids for E-sciencE INFSO-RI-508833 26 Documentation Main DPM entry point –https://twiki.cern.ch/twiki/bin/view/LCG/DataManagementTophttps://twiki.cern.ch/twiki/bin/view/LCG/DataManagementTop DPM Admin Guide –https://twiki.cern.ch/twiki/bin/view/LCG/DpmAdminGuidehttps://twiki.cern.ch/twiki/bin/view/LCG/DpmAdminGuide Troubleshooting –https://twiki.cern.ch/twiki/bin/view/LCG/LfcTroubleshootinghttps://twiki.cern.ch/twiki/bin/view/LCG/LfcTroubleshooting

27 Enabling Grids for E-sciencE INFSO-RI-508833 27 Help? Global Grid User Support (GGUS) –helpdesk@ggus.orghelpdesk@ggus.org  Your ROC will first help  If needed, DPM experts Users and administrators forum –dpm-users-forum@cern.chdpm-users-forum@cern.ch  Place to ask questions  Place to share your experience with other users/admins  To subscribe, go to http://simba.cern.chhttp://simba.cern.ch

28 Enabling Grids for E-sciencE INFSO-RI-508833 28 My message SRMv2.2 –No migration needed! –Just make sure the SRMv2.2 daemon is running Space tokens vs. User space token descriptions SRMv2.2 useful tools –dpm-reservespace –dpm-updatespace –dpm-releasespace

29 INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Questions? Sophie Lemaitre (Sophie.Lemaitre@cern.ch)Sophie.Lemaitre@cern.ch Jean-Philippe Baud (Jean-Philippe.Baud@cern.ch)Jean-Philippe.Baud@cern.ch

Download ppt "INFSO-RI-508833 Enabling Grids for E-sciencE SRMv2.2 in DPM Sophie Lemaitre Jean-Philippe."

Similar presentations

12th EELA TUTORIAL - USERS AND SYSTEM ADMINISTRATORS www.eu-eela.org E-infrastructure shared between Europe and Latin America SRM Installation and Configuration.

12th EELA TUTORIAL - USERS AND SYSTEM ADMINISTRATORS E-infrastructure shared between Europe and Latin America SRM Installation and Configuration.
DPM Name Server (DPNS) Namespace Authorization Location of physical files DPM Server Requests queuing and processing Space Management SRM Servers v1.1,

DPM Name Server (DPNS) Namespace Authorization Location of physical files DPM Server Requests queuing and processing Space Management SRM Servers v1.1,
GUMS status Gabriele Carcassi PPDG Common Project 12/9/2004.

GUMS status Gabriele Carcassi PPDG Common Project 12/9/2004.
EGEE is a project funded by the European Union under contract IST-2003-508833 Using SRM: DPM and dCache G.Donvito,V.Spinoso INFN Bari www.eu-egee.org.

EGEE is a project funded by the European Union under contract IST Using SRM: DPM and dCache G.Donvito,V.Spinoso INFN Bari
Storage: Futures Flavia Donno CERN/IT WLCG Grid Deployment Board, CERN 8 October 2008.

Storage: Futures Flavia Donno CERN/IT WLCG Grid Deployment Board, CERN 8 October 2008.
Andrew McNab - EDG Access Control - 14 Jan 2003 EU DataGrid security with GSI and Globus Andrew McNab University of Manchester

Andrew McNab - EDG Access Control - 14 Jan 2003 EU DataGrid security with GSI and Globus Andrew McNab University of Manchester
GGF Toronto 19.02.02 Spitfire A Relational DB Service for the Grid Peter Z. Kunszt European DataGrid Data Management CERN Database Group.

GGF Toronto Spitfire A Relational DB Service for the Grid Peter Z. Kunszt European DataGrid Data Management CERN Database Group.
The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.

The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.
INFSO-RI-508833 Enabling Grids for E-sciencE SRMv2.2 experience Sophie Lemaitre WLCG Workshop.

INFSO-RI Enabling Grids for E-sciencE SRMv2.2 experience Sophie Lemaitre WLCG Workshop.
DPM CCRC - 1 Research and developments DPM status and plans Jean-Philippe Baud.

DPM CCRC - 1 Research and developments DPM status and plans Jean-Philippe Baud.
LFC tutorial Jean-Philippe Baud, IT-GT, CERN July 2010.

LFC tutorial Jean-Philippe Baud, IT-GT, CERN July 2010.
StoRM Some basics and a comparison with DPM Wahid Bhimji University of Edinburgh GridPP Storage Workshop 31-Mar-101Wahid Bhimji – StoRM.

StoRM Some basics and a comparison with DPM Wahid Bhimji University of Edinburgh GridPP Storage Workshop 31-Mar-101Wahid Bhimji – StoRM.
EGEE-III INFSO-RI- 222667 Enabling Grids for E-sciencE www.eu-egee.org The Medical Data Manager : the components Johan Montagnat, Romain Texier, Tristan.

EGEE-III INFSO-RI Enabling Grids for E-sciencE The Medical Data Manager : the components Johan Montagnat, Romain Texier, Tristan.
The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.

The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.
CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t Storageware Flavia Donno CERN WLCG Collaboration Workshop CERN, 13-14 November 2008.

CERN IT Department CH-1211 Geneva 23 Switzerland t Storageware Flavia Donno CERN WLCG Collaboration Workshop CERN, November 2008.
The huge amount of resources available in the Grids, and the necessity to have the most up-to-date experimental software deployed in all the sites within.

The huge amount of resources available in the Grids, and the necessity to have the most up-to-date experimental software deployed in all the sites within.
FailSafe SGI’s High Availability Solution Mayank Vasa MTS, Linux FailSafe Gatekeeper

FailSafe SGI’s High Availability Solution Mayank Vasa MTS, Linux FailSafe Gatekeeper
INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org DPM Administration Jean-Philippe Baud (Sophie Lemaitre)

INFSO-RI Enabling Grids for E-sciencE DPM Administration Jean-Philippe Baud (Sophie Lemaitre)
EGEE-II INFSO-RI-031688 Enabling Grids for E-sciencE www.eu-egee.org EGEE middleware: gLite Data Management EGEE Tutorial 23rd APAN Meeting, Manila Jan.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE middleware: gLite Data Management EGEE Tutorial 23rd APAN Meeting, Manila Jan.
INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org ATLAS DDM Operations - III DPM at T2’s Jiří Chudoba ATLAS meeting, 25.9.2007, CNAF.

INFSO-RI Enabling Grids for E-sciencE ATLAS DDM Operations - III DPM at T2’s Jiří Chudoba ATLAS meeting, , CNAF.

Similar presentations

Ads by Google