1. Institute of Information and Communication Technologies – Bulgaria and IT
Assoc. Prof. Emanouil Atanassov
Head of Grid Technologies and Applications Department
Institute of Information and Communication Technologies – BAS

2. Historic perspective
IICT-BAS operated different types of clusters and supercomputers like the Origin 2000
IICT-BAS coordinates the Bulgarian participation in the European Grid Initiative, first through the EGEE series of projects, then EGI-InSPIRE and EGI-Engage
IICT-BAS is one of the leading partners in a series of projects that spread the expertise of advanced countries in the region to the less developed countries
Innovative scientific applications are developed and run on the available HPC equipment at IICT-BAS by teams of scientists from institutes of the Bulgarian Academy of Sciences and leading universities in Bulgaria.
Clusters at IM-BAS, INRNE-BAS and Sofia Universtity were added to the EGEE production infrastructure during EGEE and EGEE 2 projects.

3. Regional Cooperation -History
PROJECTS

4. Regional partnerships – from infrastructure to applications

5. National Roadmap for Research Infrastructures
RI: National Center for HPC and Distributed Computing - NC4HPC&DC
Financial coordinator: Ministry of Education and Science; Ministry of Transport, Information Technology and Communications.
Scientific Coordinator: Institute of Information and Communication Technologies - Bulgarian Academy of Sciences (IICT-BAS)
Consortium for supercomputers applications (since 2009 г.)
Consortium for distributed (Grid and Cloud) applications (set up , updated )
Sofia University “St Kliment Ohridski”
Institute for Nuclear Research and Nuclear Energy - Bulgarian Academy of Sciences (INRNE-BAS)
Technical University – Sofia (TU-Sofia
Medical University – Sofia (MU-Sofia)
Technical University - Gabrovo
National Institute of Geophysics, Geodesy and Geography -Bulgarian Academy of Sciences (NIGGG-BAS)
Institute of Molecular Biology “Roumen Tsanev” -Bulgarian Academy of Sciences (IMB-BAS)
Institute of Organic Chemistry with Centre for Phytochemistry - Bulgaria Academy of Sciences ( IOCCP-BAS)
Institute of Mechanics – Bulgaria Academy of Sciences (IM-BAS)
Remark: The consortium for supercomputer applications closely cooperates with Association “National Centre for Supercomputing Applications” (NCSA), which represents Bulgaria in PRACE
Institute of mathematics and Informatics – Bulgarian Academy of Sciences (IMI – BAS)

6. HPC resources in Bulgaria

7. HPC Center “Avitohol” at IICT
150 HP Cluster Platform SL250S GEN8 servers with 2 Intel Xeon E 2650 v2 CPUs and 2 Intel Xeon Phi 7120P coprocessors
Site
IICT-BAS/Avitohol
Manufacturer
Hewlett-Packard
Cores
20700
Interconnection
FDR InfiniBand
Theoretical Peak Performance
412.3 Tflop/s
RMAX Performance
264.0 Tflop/s
Memory
9600 GB
Operation System
Red Hat Enterprise Linux for HPC
Storage capacity
96 TB SAN
Top500 List on 389 place (Nov 2015) tt
Users workstations
10 Gbps

8. The Supercomputer System Avitohol at IICT-BAS
8 water-cooled racks of type HP MCS 200, paired in couples.
Each pair provides power and cooling for up to 50 kW of equipment
About 90% of the computational power comes from the accelerators – one 7120P coprocessor achieves 1.25 TFlop/s in double precision.
Total energy use at maximum load ~ 250kW.

9. The Supercomputer System Avitohol at IICT-BAS
RedHat Enterprise for HPC
Intel Cluster Studio (compilers, MPI,MKL)
Moab for resource management with web and command line interface
High-end database software - SAP HANA – in-memory database with advanced real-time analytics capabilities

10. The Supercomputer System Avitohol at IICT-BAS
150 dual-socket nodes HP SL250s Gen8 with 2 Intel Xeon E5-CPU E GHz and 2 Intel Xeon Phi 7120P coprocessors, 64 GB RAM
Interconnected with fully non-blocking FDR InfiniBand
Lustre parallel filesystem, 96 TB of disk storage

11. The Supercomputer System Avitohol at IICT-BAS
Linpack tests:
222.0 TFlop/s achieved in June, the system entered Top500 in 332nd place.
Further tuning of hardware and software produced the new result of 264 TFlop/s, maintaining presence in the November Top500 list.
For comparison the Blue Gene/P supercomputer achieved 23.9 TFlop/s and was last present in Top500 list in November 2009 in 379th place.

12. National projects supported
Project Name
Institution
PI Name
Grant
Development and Investigation of quasi-Monte Carlo Algorithms for Extreme Parallel Computer Systems, funded by Bulgarian NSF, 2014 – 2016
IICT-BAS
Assoc. Prof. T. Gurov
#DFNI-I02/8
Estimate of brown bear population in Bulgaria on the basis of mathematical, statistical and biological analysis of monitoring data”, funded by PUDOOS, 2013 – 2015
#D9190
Parallel and Distributed Computing Practices
Assoc. Prof. I. Lirkov
Numerical methods for coupled systems and computer modelling of biomedcal and environmental problems
Assoc. Prof. K. Georgiev
#DFNI-I01/5
Efficient Parallel Algorithms for Large-Scale Computational Problems, 2015 – 2016
Prof. I. Dimov
#DFNI-I02/20
Preparation of operational numerical physical model of the Black Sea
Faculty of Physics-SU
Assoc. Prof. E. Peneva
Dynamics , mechanisms and sequence of bonding of the proteins responsible for DNA repair in living cells
IMB-BAS
Assoc. Prof. S. Stoynov
#DFNI-Б02/16
Modeling of catalytic systems
Faculty of Chemistry-SU
Prof. G. Vaysilov
Efficient numerical methods and parallel algorithms for dynamical analysis of plate structures, 2016 – 2017
Assist. Prof. PhD S. Stoykov
229/
New mathematical methods of machine learning with applications in modern technology for genetic sequencing
IMI-BAS
Prof. Е. Stoimenova
#DFNI-I02/19
Fundamental research on stochastic branching processes – borderline behavior, statistical inferences and applications
FMI-SU
Prof. M. Bojkova
#DFNI-I02

13. International projects supported
Acronym
Project Name
PI Name
Grant
VI-SEEM
VRE for regional Interdisciplinary communities in Southeast Europe and the Eastern Mediterranean, 2015 – 2018
Assoc. Prof. T. Gurov
#675121, EC H2020
MMAC
Centre of Excellence for Mathematical Modeling and Advanced Computing, 2015 – 2016
Corr. Mem. S. Margenov
#664406, EC H2020
SESAME-NET
Supercomputing Expertise for SmAll and Medium Enterprise Network, 2015 – 2017
#654416, EC H2020
EGI-Engage
Engaging the EGI Community towards an Open Science Commons, 2015 – 2017
Assoc. Prof. E. Atanassov
#654142, EC H2020
CRoNoS
Computationally-intensive methods for the robust analysis of non-standard data, 2015 – 2019
Prof. A. Karaivanova
COST Action: IC1408
PRACE 4 IP
PRACE Fourth Implementation Phase, 2015 – 2017
Prof. S. Markov
#653838, EC H2020
AComIn
Advanced Computing for Innovation, 2012 – 2015
Prof. G. Angelova
#316087, EC FP7
QTLeap
Quality Translation by Deep Language Engineering Approaches, 2013 – 2016
Assoc. Prof. K. Simov
#610516, EC FP7

14. Extension HPCG cluster at IICT-BAS – BG01-IPP
36 blade servers HP BL 280c, deployed in 3 HP Cluster Platform Express 7000 enclosures, each with 2 Xeon 2.80GHz, 24 GB RAM – 576 cores total with more than 3 Tflops peak performance; 8 dual-socket HP DL 380 G6 with dual Intel 2.8 Ghz, 32 GB RAM. Non-blocking Infiniband 20 Gbps, 92% efficiency on LINPACK - 3 TFlops
Total disk storage more than 132 TB from three disk systems, interconnected with Fibre Channel.
2 HP ProLiant SL390s G7 4U servers with 16 NVIDIA Tesla M2090 graphic cards (total 8192 GPU cores with TFlops in double precision); HP SL270s Gen8 4U server with 8 Intel Xeon Phi 5110P Coprocessors (total 480 cores, 1920 threads, TFlops of double-precision peak performance per server).
Total peak theoretical peak performance TFlops

15. Organization of access
Bulgarian scientists can obtain access to the HPC systems of IICT-BAS by filling request forms that are assessed on basis of scientific merit.
Remote access through ssh:
gw.avitohol.acad.bg
Batch system: torque/moab
Development tools:
Intel Parallel Studio XE 2016
GNU Compiler Collection
MVAPICH 2.2
Open MPI
Support helpdesk:
Access policies:
More information at:

16. Organization of access
IICT-BAS maintains the BG.ACAD CA certification authority, which issues x509 certificates for use in Grids and PRACE.
BG.ACAD CA is member of EUGRID PMA organization, thus Bulgarian certificates are accepted in all European resources and the collaborating Asian-Pacific and American resources.
Avitohol is reachable through the PRACE internal network

17. Software and tools Libraries: MKL FFTW PETSc SLEPc GSL NetCDF …
Languages, tools:
R, Python, Java
cmake, gnuplot
Paraview
Scalasca
gdb, gprof

18. IICT-BAS is the Bulgarian representative in the European Grid Initiative - EGI
Data
Providers
Technology
Providers
Service Providers
All

19. European Grid Initiative - EGI
Federated services
BBMRI
DARIAH
EISCAT-3D
ELIXIR
EPOS
LifeWatch
MoBRAIN/ INSTRUCT
Environment - disaster mitigation
Innovating with RIs - The Distributed Competence Centre of EGI-Engage

20. Regional HPC and distributed computing initiative – VI-SEEM
The project VI-SEEM aims to integrate all kinds of electronic infrastructure in the region – Grid, Cloud, HPC, BigData, in order to provide single point of access for scientists in the region of South Eastern Europe and the Mediterranean.
Bulgaria is the leader in the HPC area.
Provides user-friendly integrated e-Infrastructure platform for Scientific Communities in:
Climatology,
Life Sciences,
Cultural Heritage
for the SEEM region
by linking compute, data, and visualization resources, as well as services, software and tools.
Avitohol is the most powerful supercomputer in VI-SEEM

21. VI-SEEM Geographical and historical context
VRE for regional Interdisciplinary communities in SEE and EM
Merging of SEE and EM regions
SEE: network SEEREN1-2, Grid SEE-GRID-1/2/SCI, HPC HP-SEE
EM: HPC LinkSCEEM1-2
Participant no.
Participant organisation name
Part. short name
Country
1 (Coord)
GREEK RESEARCH AND TECHNOLOGY NETWORK S.A.
GRNET
Greece 2
THE CYPRUS INSTITUTE
CyI
Cyprus 3
INSTITUTE OF INFORMATION AND COMMUNICATION TECHNOLOGIES – BULGARIAN ACADEMY OF SCIENCES
IICT-BAS
Bulgaria 4
INSTITUTE OF PHYSICS BELGRADE
IPB
Serbia 5
NATIONAL INFORMATION INFRASTRUCTURE DEVELOPMENT INSTITUTE
NIIF
Hungary 6
WEST UNIVERSITY OF TIMISOARA
UVT
Romania 7
POLYTECHNIC UNIVERSITY OF TIRANA
UPT
Albania 8
UNIVERSITY OF BANJA LUKA
UNI BL
Bosnia and Herzegovina 9
SS CYRIL AND METHODIUS UNIVERSITY OF SKOPJE
UKIM
FYR of Macedonia 10
UNIVERSITY OF MONTENEGRO
UOM
Montenegro 11
RESEARCH AND EDUCATIONAL NETWORKING ASSOCIATION OF MOLDOVA
RENAM
Moldova 12
INSTITUTE FOR INFORMATICS AND AUTOMATION PROBLEMS OF THE NATIONAL ACADEMY OF SCIENCES OF THE REPUBLIC OF ARMENIA
IIAP-NAS-RA
Armenia 13
GEORGIAN RESEARCH AND EDUCATIONAL NETWORKING ASSOCIATION
GRENA
Georgia 14
BIBLIOTHECA ALEXANDRINA BA
Egypt 15
INTER UNIVERSITY COMPUTATION CENTER
IUCC
Israel 16
SYNCHROTRON-LIGHT FOR EXPERIMENTAL SCIENCE AND APPLICATIONS IN THE MIDDLE EAST
SESAME
Jordan

22. Regional HPC and BigData Resources
Main resource providers – Bulgaria, Greece, Serbia.
Heterogeneous computing resources – CPU-only,
GPGPU-enmabled, Xeon Phi, IBM Cell
Bulgaria is the leader in computing.
Need for larger and better storage facilities

23. VI-SEEM technology context
Overall objective: Provide user-friendly integrated e-Infrastructure platform for Scientific Communities in Climatology, Life Sciences, and Digital Cultural Heritage for the SEEM region; by linking compute, data, and visualization resources, as well as services, software and tools.
Diverse computing technologies
Advent of big data
Service orientation

24. EU Network of HPC Competency Centres working with SMEs – SESAME NET
14 Partners , 13 Member States
ARCTUR, Slovenia (coordinator)
Bangor University, United Kingdom
Irish High End Computing, National University of Galway, Ireland
Fraunhofer Institute for Algorithms and Scientific Computing (SCAI), Germany
Poznan Supercomputing and Networking Center IBCH (Inst. of Bioorganic Chemistry) PAS, Poland
Greek Research and Technology Network S.A., Greece
Centro de Supercomputación de Galicia CESGA, Spain
IT4Innovations National Supercomputing Center, VŠB – Techn. University of Ostrava, Czech Republic
Yotta Advanced Computing d.o.o., Croatia
West University of Timisoara UVT, Romania
Inst. of Inform. and Comm. Technologies of BAS (IICT-BAS), Bulgaria
Ruder Bošković Institute, Croatia
Herculesstichting / Flemish Supercomputing Centre, Belgium
Vilnius University – Faculty of Mathematics and Informatics, Lithuania

25. BG05-SUGRID perspectives
The BG05-SUGrid Grid site is currently the second Bulgarian production Grid Site.
Its current hardware is rather limited, however substantial upgrade is planned:
Computing - 72 servers with dual Intel L5640 (12 cores)  total physical cores, with capability for hyperthreading
Storage - 15 servers with 24*2TB hdd each – 720 TB total

26. Conclusions
The Avitohol supercomputer is the top HPC system in Bulgaria and ensures the needs of Bulgarian scientists in the near term.
It is used mostly for HPC tasks, but some distributed computing research is also performed on it.
Grid clusters like BG01-IPP and BG05-SUGRID support diverse virtual organizations, including those from CERN.
Advanced hardware sustains involvement in both infrastructure provision and development activities
The equipment and software is upgraded regularly, following the best practices in the IT
Ensuring governmental support for the upgrade and maintenance of the infrastructure is crucial for its capability to support advanced research