Performance Troubleshooting VMware vSphere – Network

Performance Troubleshooting VMware vSphere – Network

Introduction

Networking is the fourth I/O component that I will be covering in this series of performance write ups.  Networking is another important component in the stack, if not well thought out, can lead to performance problems later down the road.  Security is an important design consideration when planning your network configuration.  One might argue that with a virtual environment your are more prone to risks since at times there is no longer a physical cabling restriction in place.  If someone has the appropriate rights in virtual center, they could bridge two logical networks together, or place a virtual machine into a DMZ.  VMware introduced vShields to mitigate your virtualized environment from some of these risks.  By creating zones you can enforce policies that can bridge, firewall, or isolate virtual machines between network segments.  When designing or upgrading your VMware environment, work closely with your network team to understand their design considerations.  If possible, leverage VLAN tagging (802.1q) to eliminate excessive physical cabling to different segments.

What’s new

Networking is a rapidly evolving technology, for most system administrators the term networking conjures up images of CAT5 UTP cables that plug into a switch.  Consider current networking technologies and you will start seeing storage and Ethernet networks beginning to merge together, which is adding more complexity to some designs. Most traditional VMware designs were originally built around fiber channel SAN’s, but now NFS and iSCSI have become an attractive solution to keep IT spend down.  Fiber Channel Over Ethernet (FCOE) is another newly introduced technology that is starting to be adopted by the industry.  This is the encapsulation of fiber channel frames over high speed (10Gb and higher) Ethernet networks.  Rather than having network cards for TCP/IP traffic and HBA’s (Host Bus Adapters) for fiber channel connectivity, administrators can implement CNA’s (Converged Network Adapters to consolidate this traffic into one card.

With the release of vSphere came some new core technologies at the network layer that expand beyond the standard VMware virtual switch.  If you purchased the enterprise plus license you can now take advantage of the Virtual Distributed Switch (vDS).  The vDS allows you to manage one central networking configuration across multiple hosts.  If you are a Cisco shop, you can take it one step further and have a complete Cisco network solution by implementing the Nexus 1000v.  The 1000v replaces the VMware vDS and gives you the same Cisco command line tools that network administrators are familiar with.

What to look for

• Check port utilization of your ESX hosts.  If you believe you are starting to experience network performance problems start with examining your host configurations.  Are you providing enough network bandwidth to your virtual machines?  Consider adding another physical network connection for the host, or upgrading to a faster connection type (i.e. gigabit or 10Gb).  Here is a great example by Kendrick Coleman of a rock solid network configuration/design:  http://www.kendrickcoleman.com/index.php?/Tech-Blog/vsphere-host-nic-configuration.html
• Follow VMware best practices.  I know this might sound obvious, but many times people grow into their environments over time and business requirements change along the way.  Review your configuration against the vSphere administrator guides.
• Implement 802.1q.  I am a fan of VLAN tagging, we have been using it since our initial implementation of VMware years ago.  You can get cable consolidation while being able to configure your virtual machines on any logical network segment you need.  This keeps costs down while using less network cables and expensive switch ports.
• Consider vShield Zones for added security.  If you are an enterprise plus customer, evaluate if vShield zones might help with security risks and network compliance, this is a feature your already paying for.
• Consider leveraging vDS as your environment grows larger.  Standard virtual switches wok fine for small to mid-sized implementations but the larger your vSphere environment grows the more automation is needed to be able to scale.  Virtual Distributed Switching allows you to maintain a central management point for your configuration.  This configuration then gets automatically “distributed” to the hosts that are part of this configuration.
• Ensure you are running the latest Virtual Hardware version 7.  With the latest version of virtual hardware comes improved drivers at the network layer.

Monitoring with Virtual Center

The first place I would start with checking network configurations is Virtual Center.  Virtual Center provides excellent reporting and gives you granular control over which metrics you would like to report against.  VMware vSphere now includes a nice graphical summary in the performance tab of the physical host.  This gives you a quick dashboard type view of the overall health of the system over a 24 hour period.  Here is the network sample:

Selecting the advance tab gives you a much more granular way of viewing performance data.  At first glance this might look overwhelming, but with a little bit of fine tuning, you can make it report on some great historical information.  Here is a snapshot of network utilization:

Check your various virtual nic’s to see if some are more overloaded than others

Monitoring with ESXTOP

Esxtop is another excellent way to monitor performance metrics on an ESX host.  Similar to the Unix/Linux “Top” command, this is designed to give an administrator a console snapshot of how the system is performing.  SSH to one of your ESX servers and execute the command “esxtop”.  The default screen that you should see is the CPU screen, if you need to monitor networking select the “n” key.  Esxtop gives you great real-time information and can even be set to log data over a longer time period, try “esxtop –a –b > performance.csv”.  Check your network connections both physical and virtual, make sure you aren’t over subscribing any particular connections or ports.

Conclusion

If you’re using VMware vSphere there are several places to go for troubleshooting networking problems.  First start out with a solid foundation, follow the best practices that VMware has spelled out in your administration guides listed above.  Work with your networking group within your company or your customers environment, make sure you are adhering to their standards and guidelines.  Consider security issues and identify and compliances that you might have to implement that might make vShields come in handy.  Leverage 802.1q if possible to simplify network configurations across all your hosts.
Look for the outliers in your environment.  If something doesn’t look right, that’s probably the case.  Scratch away at the surface and see if something pops up.  Use all possible tools available to you like PowerCLI.  Approaching problems from a different perspective will sometimes bring light to a situation you weren’t aware of.  If all else fails, engage VMware support and open a service request.  Support contracts exist for a reason and I have opened many SR’s that were new technical problems that have never been discovered by VMware support.

(Note: For Curriculum Knowledge only)