Overview of Windows 2008 HPC & InfiniBand Network

Hello All,
The purpose of this article is a quick review of HPC Cluster components and setup, Hope this helps.

  1. Windows HPC Server 2008
  2. Thing to Remember in a InfiniBand network
  3. What to expect from InfiniBand:
  4. Software Architecture for Windows 2008 HPC server
  5. Common issues with HPC 2008 Server.

Windows HPC Server 2008

Windows HPC Server 2008 is composed of a cluster of servers that includes a single head node and one or more compute nodes, the head node controls and mediates all access to the cluster resources and is the single point of management, deployment, and job scheduling for the compute cluster.

Note: You can install the client HPC pack on XP, win2k3xx machines to manage the HPC cluster.

Hardware Architecture of a HPC cluster

Best practice recommends implementing HPC Cluster on 3 different networks
  1. Application network (InfiniBand or 10 Gig Ethernet-MPI traffic)
  2. Private network (To manage the HPC cluster, PXE boot, Deployment, Management traffic)
  3. Enterprise network (connected to the corporate network)
InfiniBand provides high throughput and low latency solution compared to Ethernet.

Things to Remember in a InfiniBand network

InfiniBand:
  1. For ultimate performance customers should have a non blocking fabric (that means no contention over the InfiniBand network)
  2. Subnet manager is one of the most important components for the InfiniBand environment, this software can be installed on the switch or on a standalone machine,
    Subnet Manager is responsible for discovering all paths between InfiniBand HCA'S (Host card adapters),
    If subnet manager is not installed or enabled, the switch ports will not initialise in the first place.
    To verify if subnet manager is working and enabled, you should check if the port status is Up (from the switch Gui) and at the back of the switch the blue status light should change from blinking to static.
  1. To get logs from the switch, telnet to the switch and run the ‘Capture' command
  2. First step towards troubleshooting InfiniBand switch issues is to start Analysing the port status and symbol errors; there should be no more than 1 or 2 symbol errors per port.
  3. To install the subnet manager you need a separate licence key, If the customer has bought InfiniBand components, customer will receive a CD with a serial number, customer needs to call the Qlogic support line and provide that serial number, from which they will generate a license number for subnet manager.
  4. Drivers for the InfiniBand HBA.there are 2 types of drivers available for these HCA'S

  • Mellanox (WINOF)
    Qlogic

    From industry experience, many people have recommended to install WINOF driver rather than the Qlogic driver for optimal performance. Standard way of measuring IB performance in a HPC is to measure the MPI throughput and latency. Windows 2008 HPC server has built in tools to measure this. E.g (MPI ping pong)

What to expect from InfiniBand:

  • IB uses 2.5GB/s per link
  • SDR - 4 links - 10GB/s
  • DDR - 4 links - 20GB/s
  • DDR should translate to 900/1350MB/s bandwidth
1.5 macro second latency

In non-blocking configuration
  • IB switch has no routing capability (you need subnet manager for that)
  • Can configure LAGs (rare) no zoning or VLANs
  • No MAC/Worldwide Port Name - uses LID identifier
  • Subnet Manager contains the pathing logic



    • Can be activated on the switch (enter licence key)
    • Configured as Host machine for larger scale clusters



Software Architecture for Windows 2008 HPC server

Windows HPC Server 2008 installation involves installing the operating system on the head node, joining it to an Active Directory domain, and then installing HPC Pack 2008.

Windows HPC Server 2008 supports five different network topologies


    1. Compute nodes isolated on a private network
    2. All nodes on both public and private network
    3. Compute nodes isolated on private and application network
    4. All nodes on enterprise, private, and application networks
    5. All nodes only on enterprise network



Customer usually prefer the 3rd option, definition of a 3rd option is: The head node has three NICs, one connected to the enterprise network, one to the private network, and one to the application network.

Application network is connected to a high speed network (Gigabit Ethernet, InfiniBand) for MPI Communication.




    1. What is MPI? Message passing Interface.
    1. MS-MPI is a high-speed networking interface that runs over Gigabit Ethernet, InfiniBand, Myrinet, or any network that provides a Winsock Direct, NetworkDirect, or TCP/IP interface.
    2. MPI is used as the communication protocol for application traffic. Fundamentally, MPI is the interconnection between nodes on an HPC cluster. MPI ties nodes together.





    1. What is network direct and Winsock Direct,
    1. ND Is a new Remote Direct Memory Access (RDMA) network interface providing dramatic latency improvements for MPI applications running over high-speed fabrics
    2. The NetworkDirect protocol bypasses Windows Sockets (Winsock) and the TCP/IP stack, using Remote Direct Memory Access (RDMA) on supported hardware to improve l and reduce CPU overhead. I have pasted the diagram from Microsoft site which they use to explain ND.




    1. When you install the HBA driver, you need to install the NETWORK Direct or Winsock Direct components separately. Please refer to driver manual for instructions.
    2. If you are using the Mellanox driver package, ‘installsp -L ‘ command will list if NetworkDirect or WinsockDirect is installed on the system. If none of these components are present, HPC will use the OS TCP/IP stack for communication, this will reduce performance greatly.
    3. To verify if Network direct is being used, during the network configuration wizard the network interface used for the application network should be detected as TRUE.



Common issues with HPC Server 2008.

Depending on the problem description, gather the appropriate logs

Performance issue:
  1. My CPU is 100 % utilized, however I am not getting the performance I expected.
    Check which drivers are they using, Mellanox Winof or Qlogic, are they using network direct/ Winsock or TCP/IP.

  2. Why is my CPU not being used 100 %?
    A common place to start would be by checking underlying infrastructure configuration, if there is any contention in the MPI (application) network, this may happen. Start by analyzing application network design then troubleshooting switch issues following that up with driver verification.

  3. How do I inject my HBA driver and also install the Network Direct component via the node template generation wizard?
    You will need the INI files for the driver to do both at the same time; this operation cannot be done via the MSI setup.

Hope this helps
Huzeifa Bhai
Posted by
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)