1. 1 CHEP 2000, 10.02.2000Roberto Barbera Meeting INFN-Grid, 18.10.2000 Farming with VENUS

2. 2 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Goals of VENUS Simplify administration of heterogeneous Unix/NT clusters Increase efficiency of administration Increase productivity of users Providing a single system image Simplify software integration and installation Central point of administration Integration of Unix with WindowsNT Considering special needs in complex engineering and scientific computing environments

3. 3 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Design of todays clusters HP SGI IBM SUN Compaq DEC Compute ServersFile Servers Backup Servers Firewall WEB Email Printer Pre-, postprocessing CAD/CAE Simulations Batch, Jobflowmanagement Backup Archiv PDM Plotprocessing Databases Communication

4. 4 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Todays problems in system management I Complexity of structure: grouping of hosts, users with different working environments Complexity of data and filesystemstructures: Many projects, databases, centralized and decentralized filesystems Maintaining consistency in login behaviour: user management Fast changes in hard and software

5. 5 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Todays problems in system management II Complexity of software: many software packages used: graphical (CAD, Pre/postprocessing) sometimes big packages (>500 MB) many software releases per year, patch management licensing using different mechanisms (flexlm, netls,...) integration into the desktop (CDE, KDE,...) integration into the filesystem integration into the printer environment

6. 6 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Todays needs in system management Methods to install software and patches to be fast and reliable User management to be simple, providing best user environment for users Well defined filesystem structure to provide a global file name space Well defined operating system status (which OS, which patches) to assure quality and reliability Independence of special knowlegde of system administrators Quality driven system management instead of quick but undocumented solutions Flexibility to solve daily problems methodically

7. 7 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Single system image (SSI) I Definition: For users, an ideal cluster presents itself as a single system image, i.e. a heterogeneous workstation server cluster appears like a single big computer with many (intelligent) terminals attached. Single point of authentification Same password on any node in the cluster Unique access rights on each node Unique file space : Global file name space

8. 8 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Single system image (SSI) II Common login environment on each node common (architecture dependent) variables (e.g. $path) shell aliases desktop configuration (CDE, KDE, GNOME,...) Considering needs of users and goal to simplify the login environment Access to all printers, peripherals available for the user or user group

9. 9 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Overview of VENUS functionalities User- and file system management Uniform view of the file system /home/$user add scdel usr ch group list /$map/$key add sc del key ch list Software Management Software depot server Full documented software packages in VENUS format Pre/post/install/ remove methods and procedures VENUS tools to create VENUS software packages scsoftpkg, scsoftfile scsoftinfo sc{add|ch|del|list}pkg Configuration Management central storage of all configuration differences to original OS As global as possible as specific as necessary common architecture groups hosts VENUS methods to configure subsystems

10. 10 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 User management I scadduser, scchuser, scdeluser modify passwd map and automounter maps or corresponding information bases on NT Create, move, or delete user homes on target nodes scadduser -u calvin -h school school: /v/school/home/calvin scchuser -u calvin +h playground moves calvin from school to playground scdeluser -u calvin removes user calvin

11. 11 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 User management II Unix NT/DFS Server File Server Unix PDC,samba NT Local file system NFS SMB VENUS Master Unix: passwd map Automounter map WindowsNT: PDC configuration DFS configuration

12. 12 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Configuration management II Methods properties Methods are tasks to be executed on a client to configure a client subsystem Methods are (usually) shellscripts subsystems are: tcpip, nis, nfs, mail, resolver, xwin,cde, printer, etc. Methods are dataless, i.e. they are automatically configured in their specific context Methods are able to communicate with the configuration depot on the VENUS master Methods are able to install VENUS software packages Methods use the VENUS lookup hierarchy: host > groups > architecture> common Methods have an install and an uninstall section Methods can be modified, extended and administrators can develop their own methods

13. 13 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS architecture scrsh rshd scinstall scvenus inetd venusd Methods Context Software Depots VENUS commands Accesscontrol VENUS masterVENUS client

14. 14 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS software management I Goals: Installing Software by a simple command including configuration, licensing, pre and postinstallation Automatic maintenance of a software inventory Parallel installation of software Installing the correct software version (architecture, i.e. OS) Ability to create software packages from any 3rd party vendor Easy removal of software from a client

15. 15 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS software management II VENUS Software packages consist of: All files to be installed pre/post installation procedures and/or VENUS methods filelist, version, architecture, author Application integration Integration using wrapper scripts: setting path and environment creating symbolic links (in installation method !) wrappers are located in the standard path (e.g. /usr/local/bin)

16. 16 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 DEC VENUS software management III VENUS master Methods Context Software Depots SGI IBM SUN Compaq Software Depot HP SGI IBM SUN Compaq DEC HP Parallelized Software Installation !

17. 17 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS software management IV Commands for software management scaddpkg lsf lsfhosts This command will install LSF on each node contained in the hostgroup lsfhosts. Each client asks the VENUS master to provide the proper software package for the correct operation system type and the latest version The software package will be transfered from the nearest software depot to each client. This makes it possible to parallelize the software installation process. If the installation is finished successfully, VENUS makes an entry into the software inventory. The installation of the configuration files is done automatically in the installation method.

18. 18 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS software management V Commands for software management scdelpkg lsf lsfhosts This command will remove LSF on each node contained in the hostgroup lsfhosts. If done successfully, the entry in the software inventory is also removed. sclistpkg hostname This command creates a list of software installed on hostname and displays the current version of the software.

19. 19 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS references I DaimlerChrysler:1600 CAD Workstations in car design (SGI, HP, IBM,SUN) Main appl.: CATIA 300 CAE Workstations in engineering (SGI, IBM, HP, SUN, CRAY, NEC) 100 different applications 80 CAE Workstations in truck engineering (SGI, IBM, HP) 200 Workstations in DaimlerChrysler Research Center Stuttgart (SGI, Linux, IBM, HP) 100 Workstations in bus design and engineering (EVOBUS) (SGI,HP,IBM)

20. 20 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS references II DaimlerChrysler:Details in CAE 5 departments with different tasks: Structure mechanics, crash, engine development, various applications (> 100) High end Unix workstations and > 30 compute servers (HP, SGI Origin, NEC SX4, IBM SP) Using VENUS specials: Managing all printers with original VENUS printer method Installing complex Pre/postprocessors using VENUS methods e.g. Medina from debis, abaqus, etc. Configuring batch (LSF, NQS), Jobstarter and batchsystem using VENUS depots

21. 21 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS references III BMW:1600 CAD Workstations in car design (SGI, HP, IBM,SUN) Main appl.: CATIA 400 CAE Workstations in engineering (SGI, IBM, HP, SUN, CRAY) Compute Center with CRAY SV1, T90, Origin IBM SP, SUN HPC 100 different applications 50 Workstations at BMW Technik Gmbh (Designers of Z1)

22. 22 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS references IV 300 CAD/CAE Workstations in engineering and design (SGI, HP, IBM,SUN) 30 Workstations at BOSCH Research

23. 23 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS references V 330 CAD Workstations in ECAD design (SUN) planned: 1200 Workstation (Autumn)

24. 24 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS references VI Details ECAD at Infineon in the memory products division: >250 workstations using cadence, mentor software Big fileserver, used as file and codeserver All nodes are completely managed using VENUS methods: standard VENUS NIS, NFS, but using SUNs automounter TCPIP, DNS setup printer method modified: using lpngr Installing backup client software (legato networker) using VENUS Installing and configuring LSF using VENUS security method, SUN patch method, method to create CAD-Admin-users

25. 25 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS references VII Details cont´d Benefits: Reducing costs by increasing the efficiency for administrators: > 50 Workstations managed by one administrator Parallel execution of tasks (methods) documented and well defined status of any clients Quality driven administration in contrast to manually changing configurations Increasing efficiency of users by high stability and uptime of clients Avoiding conflicts between different administrators by VENUS multi-administrator ability: Locking of methods and central files e.g. NIS sources, passwd. Error prevention !

26. 26 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 VENUS references VIII Details cont´d Benefits: ROI (Return on investment) Service costs before using VENUS Service costs using VENUS Costs to introduce VENUS and licensing 0 3 months Time Break even

27. 27 Roberto Barbera Dipartimento di Fisica dellUniversità di Catania and INFN Catania - Italy ALICE Collaboration Meeting INFN-Grid, 18.10.2000 Costs