1. OSG Storage VDT Support and Troubleshooting Concerns Tanya Levshina

2. Support Challenges Complicated, highly distributed services Huge variety of configuration options (software and hardware) Widely diverse utilization patterns Poor error diagnostic, exception handling and propagation Lack of monitoring/diagnostic tools Support team does not have access to the service. Support personnel –Often are not authorized to use the service as user –Can not access site logs and configuration –Often can not access storage monitoring pages on the site

3. Why do we need help from ET? We need some guidance how we should organize the support efforts when Tier-2/Tier-3 sites will be in “production” mode and how to Get access to –the logs (all of them) –configuration –dCache admin interface Understand utilization patterns Establish working relationship with site storage administrators as well as VOs using storage Provide periodical training for site storage administrators to decrease the support and troubleshooting load Increase pressure on developer –to improve logging and error diagnostic –to provide service monitoring

4. Potential Solutions Maintain centralized logs (syslog-ng) –Site keeps centralized logs and allow secured web access –Site forwards logs to the central OSG log collector Maintain centralized configuration –Site provides secured web access to most configuration files Give OSG Support access to each site's dCache administrative interface. Could be done now by providing password ssh-key based access Admin interface allows to control the system – storage administrators are reluctant to allow that Encourage sites to turn on gratia storage probes – could help to understand usage patterns better Work with developers on improving logging –Started to work with dCache team –Try to apply framework for troubleshooting provided by CEDPS to dCache logs