Presentation is loading. Please wait.

Presentation is loading. Please wait.

A. Sim, CRD, L B N L 1 Data Management Foundations Workshop, Mar. 3, 2009 BeStMan and Users Alex Sim Scientific Data Management Research Group Computational.

Published byJavon Stinchfield Modified over 9 years ago

Similar presentations

Presentation on theme: "A. Sim, CRD, L B N L 1 Data Management Foundations Workshop, Mar. 3, 2009 BeStMan and Users Alex Sim Scientific Data Management Research Group Computational."— Presentation transcript:

1 A. Sim, CRD, L B N L 1 Data Management Foundations Workshop, Mar. 3, 2009 BeStMan and Users Alex Sim Scientific Data Management Research Group Computational Research Division Lawrence Berkeley National Laboratory

2 A. Sim, CRD, L B N L 2 Data Management Foundations Workshop, Mar. 3, 2009 BeStMan for Users Users have uniform access to Grid storagesUsers have uniform access to Grid storages BeStMan is one of Grid storage services Users can use the same storage clients to access BeStMan There would be no difference in accessing BeStMan even if users have experiences with other types of storage services Site with limited resourcesSite with limited resources users can run their own BeStMan

3 A. Sim, CRD, L B N L 3 Data Management Foundations Workshop, Mar. 3, 2009 What do users do with BeStMan in file requests? File System / Storage Gridftp server...... Client srmPrepareToGet/Put TURLGridFTP file transfers srmReleaseFiles/srmPutDone PUT/GET BeStMan srm-copy

4 A. Sim, CRD, L B N L 4 Data Management Foundations Workshop, Mar. 3, 2009 What do users do with BeStMan in directory operations? File System / Storage Gridftp server...... Client srmLs/srmRm/srmMkdir/srmRmdir Ls/Rm/Mkdir/Rmdir BeStMan srm-ls srm-rm srm-mkdir srm-rmdir

5 A. Sim, CRD, L B N L 5 Data Management Foundations Workshop, Mar. 3, 2009 Space Reservation Space reservationSpace reservation Storage allocation for size, time, type of storage Storage system ensures the reserved space is available Pre-allocated space, dynamic space allocation Pre-allocated spacePre-allocated space VOs or sites reserve certain portion of storage in advance Users can access the space with the pre-defined space tokens Dynamic space reservation – a.k.a. opportunistic storageDynamic space reservation – a.k.a. opportunistic storage Users can request space reservation for a certain amount of storage space for a certain amount of time (with srm-sp-reserve) Upon successful space reservation, user can access the space by the space token Dynamic space reservation on a shared resource is a big commitment Supported by some storage systems and storage sites

6 A. Sim, CRD, L B N L 6 Data Management Foundations Workshop, Mar. 3, 2009 BeStMan and Space Reservation BeStMan Full mode supportsBeStMan Full mode supports Dynamic space reservation Pre-allocated spaces with pre-defined space tokens BeStMan Gateway mode supportsBeStMan Gateway mode supports Pre-allocated spaces with pre-defined space tokens

7 A. Sim, CRD, L B N L 7 Data Management Foundations Workshop, Mar. 3, 2009 What do users do with BeStMan in space reservation? File System / Storage Gridftp server...... Client srm-sp-reserve srm-sp-update srm-sp-release Reserve/Update/Release BeStMan srm-sp-tokens srm-sp-info Space token

8 A. Sim, CRD, L B N L 8 Data Management Foundations Workshop, Mar. 3, 2009 What do users do with BeStMan in file requests with space tokens? File System / Storage Gridftp server...... Client srm-copy -spacetoken TURL GridFTP file transfers PUT/GET with space tokens BeStMan srmPrepareToGet/Put + space_token srmReleaseFiles/srmPutDone Space with space_token

9 A. Sim, CRD, L B N L 9 Data Management Foundations Workshop, Mar. 3, 2009 What do users do more with BeStMan in file requests and space reservation? File System / Storage Gridftp server...... Client srm-copy -reserve TURL GridFTP file transfers PUT/GET with space reservation for the request BeStMan srmPrepareToGet/Put + space_token srmReleaseFiles/srmPutDone srmReserveSpaceSpace token Space with space_token

10 A. Sim, CRD, L B N L 10 Data Management Foundations Workshop, Mar. 3, 2009 SRM client runs (1) Ping: srm-pingPing: srm-ping srm-ping checks the SRM server. In response to the call, SRM server returns the SRM version number as well as other backend information. srm-ping srm://hostname:port/service_handle Put: srm-copyPut: srm-copy srm-copy requests to copy files to and from SRM, between SRMs, between SRM and other storage repository, depending on the source and target URLs. srm-copy file:////local_file_path srm://hostname:port/sevice_handler\?SFN=/remotefilepath Get: srm-copyGet: srm-copy srm-copy srm://hostname:port/sevice_handler\?SFN=/remotefilepath file:////local_file_path Ls: srm-lsLs: srm-ls srm-ls srm://hostname:port/service_handle\?SFN=/file_path Rm: srm-rmRm: srm-rm srm-rm srm://hostname:port/service_handle\?SFN=/file_path Mkdir: srm-mkdirMkdir: srm-mkdir srm-mkdir srm://hostname:port/service_handle\?SFN=/dir_path Rmdir: srm-rmdirRmdir: srm-rmdir srm-rmdir srm://hostname:port/service_handle\?SFN=/dir_path

11 A. Sim, CRD, L B N L 11 Data Management Foundations Workshop, Mar. 3, 2009 SRM client runs (2) space reservation: srm-sp-reservespace reservation: srm-sp-reserve srm-sp-reserve -serviceurl srm://hostname:port/service_handle \ -size desired_total_space_size -gsize desired_guaranteed space_size -lifetime desired_space_lifetime space update: srm-sp-updatespace update: srm-sp-update srm-sp-update -serviceurl srm://hostname:port/service_handle \ -size desired_total_space_size -gsize desired_guaranteed_space_size -lifetime desired_space_lifetime -spacetoken space_token space release: srm-sp-releasespace release: srm-sp-release srm-sp-release -serviceurl srm://hostname:port/service_handle \ -spacetoken space_token space tokens retrieval: srm-sp-tokensspace tokens retrieval: srm-sp-tokens srm-sp-tokens -serviceurl srm://hostname:port/service_handle space information retrieval : srm-sp-infospace information retrieval : srm-sp-info srm-sp-info -serviceurl srm://hostname:port/service_handle -spacetoken space_token

12 A. Sim, CRD, L B N L 12 Data Management Foundations Workshop, Mar. 3, 2009 SRM client runs (3) put operation into the reserved spaceput operation into the reserved space srm-copy \ file:////local_file_path \ srm://hostname:port/service_handle\?SFN=/remote_file_path \ -spacetoken space_token purge a file from the reserved spacepurge a file from the reserved space srm-sp-purge -serviceurl srm://hostname:port/service_handle \ -spacetoken space_token -s srm://hostname:port/service_handle\?SFN=/remote_file_path

13 A. Sim, CRD, L B N L 13 Data Management Foundations Workshop, Mar. 3, 2009 How many clients can user run? How much can client machine handle?How much can client machine handle? 6-8Hz on PUT/GET operations when one client node makes 5000 requests sequentially ~23Hz on LS operations when 200 client nodes makes 5 requests sequentially Peak memory usage ~60MB during 14000 operations

14 A. Sim, CRD, L B N L 14 Data Management Foundations Workshop, Mar. 3, 2009srm-ping Ping: srm-ping srm://hostname:port/servicehandlePing: srm-ping srm://hostname:port/servicehandle E.g. srm-ping srm://dmx09.lbl.gov:6288/srm/v2/server % srm-ping srm://dmx09.lbl.gov:6288/srm/v2/server srm-ping 2.2.1.2.i2 Mon Feb 9 10:25:04 PST 2009 SRM-Clients and BeStMan Copyright(c) 2007-2009, Lawrence Berkeley National Laboratory. All rights reserved. Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:6288/srm/v2/server SRM-PING: Mon Feb 09 13:01:32 PST 2009 Calling SrmPing Request... versionInfo=v2.2 Extra information (Key=Value) backend_type=BeStMan backend_version=2.2.1.2.i2 backend_build_date=2009-02-09T18:24:42.000Z GatewayMode=Enabled gsiftpTxfServers=gsiftp://dmx09.lbl.gov clientDN=/DC=org/DC=doegrids/OU=People/CN=Alexander Sim 546622 localIDMapped=asim staticToken(0)=USATLASDATA1 desc=USATLASDATA1 size=12884901888

15 A. Sim, CRD, L B N L 15 Data Management Foundations Workshop, Mar. 3, 2009 srm-copy for PUT Put: srm-copy \ file:////local_file_path \ srm://hostname:port/servicehandle\?SFN=/remote_file_pathPut: srm-copy \ file:////local_file_path \ srm://hostname:port/servicehandle\?SFN=/remote_file_path E.g. srm-copy file:////tmp/test.data srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/mytest.file % srm-copy file:////tmp/test.data srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/mytest.file … SRM-CLIENT: Fri Aug 01 13:01:28 PDT 2008 Calling SrmPrepareToPutRequest now... request.token=asim:2(PUT) status=SRM_REQUEST_INPROGRESS explanation=null … SRM-CLIENT: Fri Aug 01 13:02:01 PDT 2008 Calling Status at Fri Aug 01 13:02:01 PDT 2008 SRM-CLIENT: Result Status from SRM (srmStatusOfPutRequest)=SRM_SUCCESS TEMPCODE=SRM_SPACE_AVAILABLE SRM-CLIENT: RemainingPinTime=868 SRM-CLIENT: received TURL=gsiftp://dmx09.lbl.gov//data/junmin/cache/asim/V.0.1-905933076/mytest.file SRM-CLIENT: Fri Aug 01 13:02:02 PDT 2008 start file transfer. SRM-CLIENT:Source=file:////tmp/test.data SRM-CLIENT:Target=gsiftp://dmx09.lbl.gov//data/junmin/cache/asim/V.0.1-905933076/mytest.file SRM-CLIENT: Fri Aug 01 13:02:05 PDT 2008 end file transfer. SRM-CLIENT: Fri Aug 01 13:02:05 PDT 2008 Calling putDone for srm://dmx09.lbl.gov:8444/srm/v2/server?SFN=/srmcache/asim/mytest.file Result.status=SRM_SUCCESS Result.Explanation=null SRM-CLIENT: Request completed with success

16 A. Sim, CRD, L B N L 16 Data Management Foundations Workshop, Mar. 3, 2009 srm-copy for GET Get: srm-copy \ srm://hostname:port/servicehandle\?SFN=/remote_file_path \ file:////local_file_pathGet: srm-copy \ srm://hostname:port/servicehandle\?SFN=/remote_file_path \ file:////local_file_path E.g. srm-copy srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir5/mytest.file file:////tmp/my.tested.data % srm-copy srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir5/mytest.file file:////tmp/my.tested.data … SRM-CLIENT: Fri Aug 01 16:48:32 PDT 2008 Connecting to httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-CLIENT: Fri Aug 01 16:48:33 PDT 2008 Calling SrmPrepareToGet Request now... request.token=asim:54(GET) status=SRM_REQUEST_INPROGRESS explanation=null StatusWaitTime=30000 SRM-CLIENT: Fri Aug 01 16:49:06 PDT 2008 Calling Status at Fri Aug 01 16:49:06 PDT 2008 Status=SRM_SUCCESS SRM-CLIENT: RemainingPinTime=9185 SRM-CLIENT: FileStatus code from server=SRM_FILE_PINNED … SRM-CLIENT: received TURL=gsiftp://dmx09.lbl.gov//tmp/junmin/gt3/cache/asim/V.2-1870486411/mytest.file SRM-CLIENT: Fri Aug 01 16:49:07 PDT 2008 start file transfer. SRM-CLIENT:Source=gsiftp://dmx09.lbl.gov//tmp/junmin/gt3/cache/asim/V.2-1870486411/mytest.file SRM-CLIENT:Target=file:////tmp/my.tested.data SRM-CLIENT: Fri Aug 01 16:49:10 PDT 2008 end file transfer. SRM-CLIENT: Request completed with success

17 A. Sim, CRD, L B N L 17 Data Management Foundations Workshop, Mar. 3, 2009srm-ls Ls: srm-ls srm://hostname:port/servicehandle\?SFN=/file_pathLs: srm-ls srm://hostname:port/servicehandle\?SFN=/file_path E.g. srm-ls srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir/mytest.file % srm-ls srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir/mytest.file … SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-DIR: Fri Aug 01 13:23:29 PDT 2008 Calling srmLsRequest SRM-DIR:.......................... Status : SRM_SUCCESS Explanation : Ref:asim:9(LS) Request token=null SURL=/srmcache/asim/testdir/mytest.file Bytes=15 FileType=FILE StorageType=VOLATILE Status=SRM_SUCCESS Explanation=Read from disk OwnerPermission=null LifetimeLeft=null LifetimeAssigned=null CheckSumType=null CheckSumValue=null FileLocality=null OwnerPermission=null GroupPermission=null OtherPermission=null ArrayOfSpaceTokens=null getRetentionPolicyInfo=null getLastModificationTime=null CreatedAtTime=Fri Aug 01 13:20:43 PDT 2008

18 A. Sim, CRD, L B N L 18 Data Management Foundations Workshop, Mar. 3, 2009srm-rm Rm: srm-rm srm://hostname:port/servicehandle\?SFN=/file_pathRm: srm-rm srm://hostname:port/servicehandle\?SFN=/file_path E.g. srm-rm srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir6/mytest2.file % srm-rm srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir6/mytest2.file srm-rm SRM-Clients and BeStMan Copyright(c) 2007-2008, Lawrence Berkeley National Laboratory. All rights reserved. Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-DIR: Fri Aug 01 14:11:24 PDT 2008 Calling SrmRmFile SRM-DIR: Total files to remove: 1 status=SRM_SUCCESS explanation=null surl=srm://dmx09.lbl.gov:8444/srm/v2/server?SFN=/srmcache/asim/testdir6/mytest2.file status=SRM_SUCCESS explanation=null

19 A. Sim, CRD, L B N L 19 Data Management Foundations Workshop, Mar. 3, 2009srm-mkdir Mkdir: srm-mkdir srm://hostname:port/servicehandle\?SFN=/dir_pathMkdir: srm-mkdir srm://hostname:port/servicehandle\?SFN=/dir_path E.g. srm-mkdir srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir % srm-mkdir srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir srm-mkdir SRM-Clients and BeStMan Copyright(c) 2007-2008, Lawrence Berkeley National Laboratory. All rights reserved. Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-DIR: Fri Aug 01 13:20:00 PDT 2008 Calling SrmMkdir status=SRM_SUCCESS explanation=null

20 A. Sim, CRD, L B N L 20 Data Management Foundations Workshop, Mar. 3, 2009srm-rmdir Rmdir: srm-rmdir srm://hostname:port/servicehandle\?SFN=/dir_pathRmdir: srm-rmdir srm://hostname:port/servicehandle\?SFN=/dir_path E.g. srm-rmdir srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir6 % srm-rmdir srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir6 srm-rmdir SRM-Clients and BeStMan Copyright(c) 2007-2008, Lawrence Berkeley National Laboratory. All rights reserved. Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-DIR: Fri Aug 01 14:12:57 PDT 2008 Calling SrmRmdir SRM-DIR:........................ status=SRM_SUCCESS explanation=null

21 A. Sim, CRD, L B N L 21 Data Management Foundations Workshop, Mar. 3, 2009srm-sp-reserve space reservation: srm-sp-reservespace reservation: srm-sp-reserve srm-sp-reserve -serviceurl srm://hostname:port/service_handle \ -size desired_total_space_size -gsize desired_guaranteed space_size -lifetime desired_space_lifetime E.g. srm-sp-reserve srm://dmx09.lbl.gov:8444/srm/v2/server -size 50000 -gsize 40000 -lifetime 3600 % srm-sp-reserve srm://dmx09.lbl.gov:8444/srm/v2/server -size 50000 -gsize 40000 -lifetime 3600 srm-sp-reserve SRM-Clients and BeStMan Copyright(c) 2007-2008, Lawrence Berkeley National Laboratory. All rights reserved. Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-CLIENT: Mon Aug 04 14:56:43 PDT 2008 Calling SrmReserveSpace SRM-CLIENT:....................... Status Code=SRM_SUCCESS SpaceToken=V.4 TotalReservedSpaceSize=50000 Guaranteed Space Size=50000 Lifetime=3600

22 A. Sim, CRD, L B N L 22 Data Management Foundations Workshop, Mar. 3, 2009srm-sp-update space update: srm-sp-updatespace update: srm-sp-update $SRM_HOME/bin/srm-sp-update -serviceurl srm://hostname:port/service_handle \ -size desired_total_space_size -gsize desired_guaranteed_space_size -lifetime desired_space_lifetime -spacetoken space_token E.g. srm-sp-update srm://dmx09.lbl.gov:8444/srm/v2/server -spacetoken V.8 -size 30000 -gsize 25000 -lifetime 6000 % srm-sp-update srm://dmx09.lbl.gov:8444/srm/v2/server -spacetoken V.8 -size 30000 -gsize 25000 -lifetime 6000 srm-sp-update SRM-Clients and BeStMan Copyright(c) 2007-2008, Lawrence Berkeley National Laboratory. All rights reserved. Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-CLIENT: Fri Aug 01 18:10:33 PDT 2008 Calling updateSpace request status=SRM_SUCCESS Request token=null lifetime=5999 Min=30000 Max=30000

23 A. Sim, CRD, L B N L 23 Data Management Foundations Workshop, Mar. 3, 2009srm-sp-release space release: srm-sp-releasespace release: srm-sp-release srm-sp-release -serviceurl srm://hostname:port/service_handle -spacetoken space_token E.g. srm-sp-release srm://dmx09.lbl.gov:8444/srm/v2/server -spacetoken V.3 % srm-sp-release srm://dmx09.lbl.gov:8444/srm/v2/server -spacetoken V.3 srm-sp-release SRM-Clients and BeStMan Copyright(c) 2007-2008, Lawrence Berkeley National Laboratory. All rights reserved. Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-CLIENT: Fri Aug 01 17:57:58 PDT 2008Calling ReleaseSpace request.... SRM-CLIENT: Releasing space for token=V.3 status=SRM_SUCCESS

24 A. Sim, CRD, L B N L 24 Data Management Foundations Workshop, Mar. 3, 2009srm-sp-tokens space tokens retrieval: srm-sp-tokensspace tokens retrieval: srm-sp-tokens srm-sp-tokens -serviceurl srm://hostname:port/service_handle E.g. srm-sp-tokens srm://dmx09.lbl.gov:8444/srm/v2/server -userdesc my_space_test_1 % srm-sp-tokens srm://dmx09.lbl.gov:8444/srm/v2/server -userdesc my_space_test_1 … SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-CLIENT: Fri Aug 01 17:49:49 PDT 2008 Calling srmGetSpaceTokens... SRM-CLIENT:................................... Status=SRM_SUCCESS Explanation=null SRM-CLIENT (0)SpaceToken=V.5 % srm-sp-tokens srm://dmx09.lbl.gov:8444/srm/v2/server … SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-CLIENT: Fri Aug 01 17:49:49 PDT 2008 Calling srmGetSpaceTokens... SRM-CLIENT:................................... Status=SRM_SUCCESS Explanation=null SRM-CLIENT (0)SpaceToken=V.2 SRM-CLIENT (1)SpaceToken=V.3 SRM-CLIENT (2)SpaceToken=V.4 SRM-CLIENT (3)SpaceToken=V.5 SRM-CLIENT (4)SpaceToken=V.6

25 A. Sim, CRD, L B N L 25 Data Management Foundations Workshop, Mar. 3, 2009srm-sp-info space information retrieval : srm-sp-infospace information retrieval : srm-sp-info srm-sp-info -serviceurl srm://hostname:port/service_handle -spacetoken space_token E.g. srm-sp-info srm://dmx09.lbl.gov:8444/srm/v2/server -spacetoken V.8 % srm-sp-info srm://dmx09.lbl.gov:8444/srm/v2/server -spacetoken V.8 … SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-CLIENT: Fri Aug 01 17:54:26 PDT 2008 Calling SrmGetSpaceMetaDataRequest... SRM-CLIENT:....space token details.... status=SRM_SUCCESS SpaceToken=V.8 TotalSize=50000 Owner=asim LifetimeAssigned=3600 LifetimeLeft=3528 UnusedSize=50000 GuaranteedSize=50000 RetentionPolicy=REPLICA AccessLatency=ONLINE status=SRM_SUCCESS explanation=my_test-space_1

26 A. Sim, CRD, L B N L 26 Data Management Foundations Workshop, Mar. 3, 2009 srm-copy with space token put operation into the reserved spaceput operation into the reserved space srm-copy \ file:////local_file_path \ srm://hostname:port/service_handle\?SFN=/remote_file_path \ -spacetoken space_token E.g. srm-copy file:////tmp/test.data srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/mytest.file -spacetoken V.2 % srm-copy file:////tmp/test.data srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/mytest.file -spacetoken V.2 … SRM-CLIENT: Mon Aug 04 12:25:38 PDT 2008 Connecting to httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-CLIENT: Mon Aug 04 12:25:39 PDT 2008 Calling SrmPrepareToPutRequest now... request.token=asim:1(PUT) status=SRM_REQUEST_INPROGRESS explanation=null SRM-CLIENT: Next status call in 30 seconds. SRM-CLIENT: Mon Aug 04 12:26:12 PDT 2008 Calling Status at Mon Aug 04 12:26:12 PDT 2008 SRM-CLIENT: Result Status from SRM (srmStatusOfPutRequest)=SRM_SUCCESS STATUSCODE=SRM_SPACE_AVAILABLE SRM-CLIENT: RemainingPinTime=868 SRM-CLIENT: received TURL=gsiftp://dmx09.lbl.gov//tmp/junmin/gt3/cache/asim/V.2-57524824/mytest.file SRM-CLIENT: Mon Aug 04 12:26:13 PDT 2008 start file transfer. SRM-CLIENT:Source=file:////tmp/test.data SRM-CLIENT:Target=gsiftp://dmx09.lbl.gov//tmp/junmin/gt3/cache/asim/V.2-57524824/mytest.file SRM-CLIENT: Mon Aug 04 12:26:16 PDT 2008 end file transfer. SRM-CLIENT: Mon Aug 04 12:26:16 PDT 2008 Calling putDone for srm://dmx09.lbl.gov:8444/srm/v2/server?SFN=/srmcache/asim/mytest.file Result.status=SRM_SUCCESS Result.Explanation=null SRM-CLIENT: Request completed with success

27 A. Sim, CRD, L B N L 27 Data Management Foundations Workshop, Mar. 3, 2009srm-sp-purge purge a file from the reserved spacepurge a file from the reserved space srm-sp-purge -serviceurl srm://hostname:port/service_handle \ -spacetoken space_token -s srm://hostname:port/service_handle\?SFN=/remote_file_path E.g. srm-sp-purge srm://dmx09.lbl.gov:8444/srm/v2/server -spacetoken V.4 \ -s srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir6/mytest.file % srm-sp-purge srm://dmx09.lbl.gov:8444/srm/v2/server -spacetoken V.4 \ -s srm://dmx09.lbl.gov:8444/srm/v2/server\?SFN=/srmcache/asim/testdir6/mytest.file srm-sp-purge SRM-Clients and BeStMan Copyright(c) 2007-2008, Lawrence Berkeley National Laboratory. All rights reserved. Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman SRM-CLIENT: Connecting to serviceurl httpg://dmx09.lbl.gov:8444/srm/v2/server SRM-CLIENT: Fri Aug 01 18:03:33 PDT 2008 Calling purgeFromSpace status=SRM_SUCCESS explanation=null SURL=srm://dmx09.lbl.gov:8444/srm/v2/server?SFN=/srmcache/asim/testdir6/mytest.file status=SRM_SUCCESS explanation=null

28 A. Sim, CRD, L B N L 28 Data Management Foundations Workshop, Mar. 3, 2009Summary BeStMan is one of SRMs and supported by OSGBeStMan is one of SRMs and supported by OSG Users have uniform access to BeStMan and other Grid storages There would be no difference in accessing BeStMan and other Grid Storages even if there are different types of storage services OSG supports users!OSG supports users! osg-storage@opensciencegrid.orgosg-storage@opensciencegrid.org

29 A. Sim, CRD, L B N L 29 Data Management Foundations Workshop, Mar. 3, 2009 Extra

30 A. Sim, CRD, L B N L 30 Data Management Foundations Workshop, Mar. 3, 2009 BeStMan Server Installation Samples

31 A. Sim, CRD, L B N L 31 Data Management Foundations Workshop, Mar. 3, 2009Documents OSG documents describe step-by-step instructionsOSG documents describe step-by-step instructions https://twiki.grid.iu.edu/bin/view/Documentation/BestmanGateway https://twiki.grid.iu.edu/bin/view/Documentation/BestmanGateway-Xrootd

32 A. Sim, CRD, L B N L 32 Data Management Foundations Workshop, Mar. 3, 2009Preparation Grid service certificateGrid service certificate GridFTP server hostname, port numberGridFTP server hostname, port number Default: $hostname:2811 GLOBUS_TCP_PORT_RANGEGLOBUS_TCP_PORT_RANGE If $hostname has a firewall Two open port numbers to be assigned to BeStManTwo open port numbers to be assigned to BeStMan Default: 8080, 8443 Local disk path and size information to be managed by BeStManLocal disk path and size information to be managed by BeStMan Log file path informationLog file path information Default: /var/log Java 1.6.0_x installation pathJava 1.6.0_x installation path Default from $PATH BeStMan installation directoryBeStMan installation directory Default: current directory BeStMan downloads from http://datagrid.lbl.gov/bestmanBeStMan downloads from http://datagrid.lbl.gov/bestman Or VDT pacman installation

33 A. Sim, CRD, L B N L 33 Data Management Foundations Workshop, Mar. 3, 2009 Preparation : example Grid service certificateGrid service certificate /DC=org/DC=doegrids/OU=Services/CN=srmdemo/dmx09.lbl.gov $HOME/osgdemo/srmdemocert.pem and $HOME/osgdemo/srmdemokey.pem GridFTP server hostname, port numberGridFTP server hostname, port number dmx09.lbl.gov GLOBUS_TCP_PORT_RANGEGLOBUS_TCP_PORT_RANGE 6201,6299 Two open port numbers to be assigned to BeStManTwo open port numbers to be assigned to BeStMan 6249, 6250 Local disk path and size information to be managedLocal disk path and size information to be managed /data/osg/demo/data : 20000MB Log file path informationLog file path information /data/osg/demo/log Java 1.6.0_x installation pathJava 1.6.0_x installation path /software/jdk1.6.0_12 BeStMan installation directoryBeStMan installation directory /data/osg/bestman

34 A. Sim, CRD, L B N L 34 Data Management Foundations Workshop, Mar. 3, 2009Configuration./configure \./configure \ --with-globus-tcp-port-range=6201,6299 \ --with-http-port=6249 \ --with-https-port=6250 \ --with-eventlog-path=/data/osg/demo/log \ --with-cachelog-path=/data/osg/demo/log \ --with-replica-storage-path=/data/osg/demo/data \ --with-replica-storage-size=20000 \ --with-certfile-path=/home/users/asim/osgdemo/srmcert.pem \ --with-keyfile-path=/home/users/asim/osgdemo/srmkey.pem

35 A. Sim, CRD, L B N L 35 Data Management Foundations Workshop, Mar. 3, 2009 BeStMan server start/stop As a user/service loginAs a user/service login $SRM_HOME/sbin/SXXbestman.personal start $SRM_HOME/sbin/bestman.server >& /tmp/bestman-.log 2>&1 & $SRM_HOME/sbin/bestman.server $SRM_HOME/sbin/SXXbestman.personal stop As a root loginAs a root login $SRM_HOME/sbin/SXXbestman start su $SRMOWNER -c "$SRM_HOME/sbin/bestman.server &“ $SRM_HOME/sbin/ SXXbestman stop

Download ppt "A. Sim, CRD, L B N L 1 Data Management Foundations Workshop, Mar. 3, 2009 BeStMan and Users Alex Sim Scientific Data Management Research Group Computational."

Similar presentations

DPM Name Server (DPNS) Namespace Authorization Location of physical files DPM Server Requests queuing and processing Space Management SRM Servers v1.1,

DPM Name Server (DPNS) Namespace Authorization Location of physical files DPM Server Requests queuing and processing Space Management SRM Servers v1.1,
1 SRM-Lite: overcoming the firewall barrier for large scale file replication Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory April, 2007.

1 SRM-Lite: overcoming the firewall barrier for large scale file replication Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory April, 2007.
– n° 1 StoRM latest performance test results Alberto Forti Otranto, Jun 8 2006.

– n° 1 StoRM latest performance test results Alberto Forti Otranto, Jun
Module 4: Configuring Caching. Overview Cache Overview Configuring Cache Policy Configuring Cache Settings Configuring Scheduled Content Downloads.

Module 4: Configuring Caching. Overview Cache Overview Configuring Cache Policy Configuring Cache Settings Configuring Scheduled Content Downloads.
A. Sim, CRD, L B N L 1 Data Management Foundations Workshop, Mar. 3, 2009 Storage in OSG and BeStMan Alex Sim Scientific Data Management Research Group.

A. Sim, CRD, L B N L 1 Data Management Foundations Workshop, Mar. 3, 2009 Storage in OSG and BeStMan Alex Sim Scientific Data Management Research Group.
A crash course in njit’s Afs

A crash course in njit’s Afs
1 Web Server Administration Chapter 3 Installing the Server.

1 Web Server Administration Chapter 3 Installing the Server.
OSG End User Tools Overview OSG Grid school – March 19, 2009 Marco Mambelli - University of Chicago A brief summary about the system.

OSG End User Tools Overview OSG Grid school – March 19, 2009 Marco Mambelli - University of Chicago A brief summary about the system.
Nimbus & OpenNebula Young Suk Moon. Nimbus - Intro Open source toolkit Provides virtual workspace service (Infrastructure as a Service) A client uses.

Nimbus & OpenNebula Young Suk Moon. Nimbus - Intro Open source toolkit Provides virtual workspace service (Infrastructure as a Service) A client uses.
Www.epikh.eu The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) Overview of software tools for gLite installation & configuration.

The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) Overview of software tools for gLite installation & configuration.
A. Sim, CRD, L B N L GIN-Data : SRM Island Inter-Op Testing With SRM-TESTER Alex Sim, Vijaya Natarajan Computational Research Division Lawrence Berkeley.

A. Sim, CRD, L B N L GIN-Data : SRM Island Inter-Op Testing With SRM-TESTER Alex Sim, Vijaya Natarajan Computational Research Division Lawrence Berkeley.
SRM at Clemson Michael Fenn. What is a Storage Element? Provides grid-accessible storage space. Is accessible to applications running on OSG through either.

SRM at Clemson Michael Fenn. What is a Storage Element? Provides grid-accessible storage space. Is accessible to applications running on OSG through either.
A. Sim, CRD, L B N L 1 US CMS Workshop, Mar. 3, 2009 Berkeley Storage Manager (BeStMan) Alex Sim Scientific Data Management Research Group Computational.

A. Sim, CRD, L B N L 1 US CMS Workshop, Mar. 3, 2009 Berkeley Storage Manager (BeStMan) Alex Sim Scientific Data Management Research Group Computational.
OSG Site Provide one or more of the following capabilities: – access to local computational resources using a batch queue – interactive access to local.

OSG Site Provide one or more of the following capabilities: – access to local computational resources using a batch queue – interactive access to local.
CERN, 29 August 2006 Status Report Riccardo Zappi INFN-CNAF, Bologna.

CERN, 29 August 2006 Status Report Riccardo Zappi INFN-CNAF, Bologna.
The Disk Resource Manager A Berkeley SRM for Disk Cache Implementation and Testing Experience on the OSG Jorge Luis Rodriguez IBP/Grid Mini-Workshop Apr.

The Disk Resource Manager A Berkeley SRM for Disk Cache Implementation and Testing Experience on the OSG Jorge Luis Rodriguez IBP/Grid Mini-Workshop Apr.
A. Sim, CRD, L B N L 1 Oct. 23, 2008 BeStMan Extra Slides.

A. Sim, CRD, L B N L 1 Oct. 23, 2008 BeStMan Extra Slides.
SDM Center February 2, 2005 Progress on MPI-IO Access to Mass Storage System Using a Storage Resource Manager Ekow J. Otoo, Arie Shoshani and Alex Sim.

SDM Center February 2, 2005 Progress on MPI-IO Access to Mass Storage System Using a Storage Resource Manager Ekow J. Otoo, Arie Shoshani and Alex Sim.
Status report on SRM v2.2 implementations: results of first stress tests 2 th July 2007 Flavia Donno CERN, IT/GD.

Status report on SRM v2.2 implementations: results of first stress tests 2 th July 2007 Flavia Donno CERN, IT/GD.
A. Sim, CRD, L B N L 1 OSG Applications Workshop 6/1/2005 OSG SRM/DRM Readiness and Plan Alex Sim / Jorge Rodriguez Scientific Data Management Group Computational.

A. Sim, CRD, L B N L 1 OSG Applications Workshop 6/1/2005 OSG SRM/DRM Readiness and Plan Alex Sim / Jorge Rodriguez Scientific Data Management Group Computational.

Similar presentations

Ads by Google