Interconnect

Gigabit Ethernet Network

The Ethernet network interconnects all the entities of the cluster and is used for administrative tasks (installation, management, supervision, NFS, PBS communications,...).

Infiniband Network

The infiniband network is made of  63 switches: 1 per chassis, 10 Voltaire QDR ISR-036 and 20 Mellanox QDR IS5030 switches.

The infiniband network is composed of 3 levels: the Leaf level (L1), the Core level (L2) and the Top level (L3).

The higher level, L3, connects 2 non-blocking fat-tree which include:

• FT1: 14 Haswell nodes chassis;
• FT2: 19 Ivy Bridge nodes chassis + frontal nodes + GPFS nodes.

Inside each fat tree, communication between entities are non-blocking and between the two fat-trees, communications are blocking. The link between the L3 switches has been designed to satisfy the GPFS traffic but not to support computation between the two blocks.