Principal Engineer Ravi Soundararajan explains the concepts behind the Enhanced LACP support in vSphere Distributed Switch 5.5.
vSphere network I/O control version 3, available in vSphere 6.0, offers granular network resource reservation and allocation across the entire switch.
VMware vSphere® virtual machine encryption (VM encryption) is a feature introduced in vSphere 6.5 to enable the encryption of virtual machines. VM encryption provides security to VMDK data by encrypting I/Os from a virtual machine (which has the VM encryption feature enabled) before it gets stored in the VMDK. In this paper, we quantify the impact of using VM encryption on a VM’s I/O performance as well as on some of the VM provisioning operations like VM clone, power-on, and snapshot creation. We show that while VM encryption can lead to bottlenecks in I/O throughput and latency for ultra-high-performance devices (like a high-end NVMe drive) that can support hundreds of thousands of IOPS, for most regular types of storage, like enterprise class SSD or VMware vSAN™, the impact on I/O performance is very minimal.
VM encryption supports the encryption of virtual machine files, virtual disk files, and core dump files. Some of the files associated with a virtual machine like log files, VM configuration files, and virtual disk descriptor files are not encrypted. This is because they mostly contain non-sensitive data and operations like disk management should be supported whether or not the underlying disk files are secured. VM encryption uses vSphere APIs for I/O filtering (VAIO), henceforth referred to as IOFilter.
IOFilter is an ESXi framework that allows the interception of VM I/Os in the virtual SCSI emulation (VSCSI) layer. On a high level, the VSCSI layer can be thought of as the layer in ESXi just below the VM and above the VMFS file system. The IOFilter framework enables developers, both VMware and third party vendors, to write filters to implement more services using VM I/Os like encryption, caching, and replication. This framework is implemented entirely in user space. This allows the VM I/Os to be isolated cleanly from the core architecture of ESXi, thereby eliminating any potential issues to the core functionality of the hypervisor. In case of any failure, only the VM in question would be affected. There can be multiple filters enabled for a particular VM or a VMDK, and these filters are typically chained in a manner shown below, so that I/Os are processed by each of these filters serially, one after the other, and then finally either passed down to VMFS or completed within one of the filters. This is illustrated in Figure 1.
VM Encryption Overview
The primary purpose of VM encryption is to secure the data in VMDKs, such that when the VMDK data is accessed by any unauthorized entity, it gets only meaningless data. The VM that legitimately owns the VMDK has the necessary key to decrypt the data whenever read and then fed to the guest operating system. This is done using industry-standard encryption algorithms to secure this traffic with minimal overhead.
While VM encryption does not impose any new hardware requirements, using a processor that supports the AES-NI instruction set would speed up the encryption/decryption operation. In order to quantify the performance expectations on a traditional server without an AES-NI enabled processor, the results in this paper are from slightly older servers that do not support the AES-NI instruction set.
Figure 2 shows the various components involved as part of the VM encryption mechanism. It consists of an external key management server (KMS), the vCenter Server system, and an ESXi host or hosts. vCenter Server requests keys from an external KMS, which generates and stores the keys and passes them down to vCenter Server for distribution. An important aspect to note is that there is no “per-block hashing” for the virtual disk.
This means, VM encryption provides data protection against snooping and not against data corruption since there is no hash for detecting corruption and recovering from it. For more security, the encryption takes into account not only the encryption key, but also the block’s address. This means two blocks of a VMDK with the same content encrypt to different data.
To visualize the mechanism of encryption (and decryption), we need to look at how the various elements in the security policy are laid out topologically. The KMS is the central server in this security-enabled landscape. Figure 3 shows a simplified topology.
The KMS is a secure centralized repository of cryptographic keys. There can be more than one KMS configured with a vCenter Server. However, they need to be configured such that only KMSs that replicate keys between themselves (usually from the same vendor) should be added to the same KMS cluster. Otherwise each KMS should be added under a different KMS cluster. One of the KMS clusters must be designated as the default in vCenter Server. Only Key Management Interoperability Protocol (KMIP) v1.1 compliant KMSs are supported and vCenter Server is the client of KMS. Using KMIP enables vCenter Server to talk to any KMIP compliant KMS vendor. Before transacting with the KMS, vCenter Server must establish a trust connection with it, which needs to be done manually.
Download a full VMware vSphere Virtual Machine Encryption Performance vSphere 6.5 Guide.
This video provides an overview of vSphere Network I/O Control. Network I/O Control allows you to manage traffic on your distributed switch by dividing it into predefined or user defined network groups. You can tailor traffic bandwidth to the specific needs of your network.
With the rise in popularity of hybrid cloud computing, where VM-sensitive data leaves the traditional IT environment and traverses over the public networks, IT administrators and architects need a simple and secure way to protect critical VM data that traverses across clouds and over long distances.
The Encrypted vMotion feature available in VMware vSphere® 6.5 addresses this challenge by introducing a software approach that provides end-to-end encryption for vMotion network traffic. The feature encrypts all the vMotion data inside the vmkernel by using the most widely used AES-GCM encryption standards, and thereby provides data confidentiality, integrity, and authenticity even if vMotion traffic traverses untrusted network links.
Experiments conducted in the VMware performance labs using industry-standard workloads show the following:
- vSphere 6.5 Encrypted vMotion performs nearly the same as regular, unencrypted vMotion.
- The CPU cost of encrypting vMotion traffic is very moderate, thanks to the performance optimizations added to the vSphere 6.5 vMotion code path.
- vSphere 6.5 Encrypted vMotion provides the proven reliability and performance guarantees of regular, unencrypted vMotion, even across long.
VMware vSphere® vMotion®  provides the ability to migrate a running virtual machine from one vSphere host to another, with no perceivable impact to the virtual machine’s performance. vMotion brings enormous benefits to administrators—it reduces server downtime and facilitates automatic load-balancing.
During migration, the entire memory and disk state associated with a VM, along with its metadata, are transferred over the vMotion network. It is possible during VM migration for an attacker with sufficient network privileges to compromise a VM by modifying its memory contents during the transit to subvert the VM’s applications or its guest operating system. Due to this possible security risk, VMware highly recommended administrators use an isolated or secured network for vMotion traffic, separate from other datacenter networks such as the management network or provisioning network. This protected the VM’s sensitive data as it traversed over a secure network.
Even though this recommended approach adds slightly higher network and administrative complexity, it works well in a traditional IT environment where the customer owns the complete network infrastructure and can secure it. In a hybrid cloud, however, workloads move dynamically between clouds and datacenters over secured and unsecured network links. Therefore, it is essential to secure sensitive vMotion traffic at the network endpoints. This protects critical VM data even as the vMotion traffic leaves the traditional IT environment and traverses over the public networks.
vSphere 6.5 introduces Encrypted vMotion, which provides end-to-end encryption of vMotion traffic and protects VM data from eavesdropping occurrences on untrusted network links. Encrypted vMotion provides complete confidentiality, integrity, and authenticity of the data transferred over a vMotion network without any requirement for dedicated networks or additional hardware.
The sections that follow describe:
- vSphere 6.5 Encrypted vMotion technology and architecture
- How to configure Encrypted vMotion from the vSphere Client
- Performance implications of encrypting vMotion traffic using real-life workload scenarios
- Best practices for deployment
Encrypted vMotion Architecture
vMotion uses TCP as the transport protocol for migrating the VM data. To secure VM migration, vSphere 6.5 encrypts all the vMotion traffic, including the TCP payload and vMotion metadata, using the most widely used AES-GCM encryption standard algorithms, provided by the FIPS-certified vmkernel vmkcrypto module.
Encrypted vMotion does not rely on the Secure Sockets Layer (SSL) or Internet Protocol Security (IPsec) technologies for securing vMotion traffic. Instead, it implements a custom encrypted protocol above the TCP layer. This is done primarily for performance, but also for reasons explained below.
SSL is compute intensive and completely implemented in user space, while vMotion, which constitutes core ESXi, executes in kernel space. This means, if vMotion were to rely on SSL, each encryption/decryption call would need to traverse across kernel and user spaces, thereby resulting in excessive performance overhead. Using the encryption algorithms provided by the vmkernel vmkcrypto module enables vMotion to avoid such a performance penalty.
Although IPSec can be used to secure vMotion traffic, its usability is limited in the vSphere environment because ESXi hosts support IPsec only for IPv6 traffic, but not for IPv4 traffic. Besides, implementing a custom protocol above the TCP layer gives vMotion the ability to create the appropriate number of vMotion worker threads, and coordinate efficiently among them to spread the encryption/decryption CPU load across multiple cores.
Download a full VMware vSphere Encrypted vMotion Architecture, Performance and Best Practices Study.
VMware vSphere® Distributed Resource Scheduler™ (DRS) is more than a decade old and is constantly innovating with every new version. In vSphere 6.5, DRS comes with many new features and performance improvements to ensure more efficient load balancing and VM placement, faster response times, and simplified cluster management.
In this paper, we cover some of the key features and performance improvements to highlight the more efficient, faster, and lighter DRS in vSphere 6.5.
Historically, vSphere DRS has been reactive—it reacts to any changes in VM workloads and migrates the VMs to distribute load across different hosts. In vSphere 6.5, with VMware vCenter Server® working together with VMware vRealize® Operations™ (vROps), DRS can act upon predicted future changes in workloads. This helps DRS migrate VMs proactively and makes room in the cluster to accommodate future workload demand.
For example, if your VMs’ workload is going to spike at 9am every day, predictive DRS will be able to detect this pattern before-hand based on historical data from vROPs, and can prepare the cluster resources by using either of the following techniques:
- Migrating the VMs to different hosts to accommodate the future workload and avoid host over-commitment
- Bringing back a new host from stand-by mode using VMware vSphere® Distributed Power Management™ (DPM) to accommodate the future demand
How It Works
To enable predictive DRS, you need to link vCenter Server to a vROps instance (that supports predictive DRS), which monitors the resource usage pattern of VMs and generates predictions. Once vROps starts monitoring VM workloads, it generates predictions after a specified learning period. The generated predictions are then provided to vCenter Server for DRS to consume.
Once the VMs’ workload predictions are available, DRS evaluates the demand of a VM based on its current resource usage and predicted future resource usage.
- Demand of a VM = Max (current usage, predicted future usage)
Considering the maximum of current and future resource usage ensures that DRS does not clip any VM’s current demand in favor of its future demand. For the VMs which do not have predictions, DRS computes resource
demand based on only the current resource usage.
Look Ahead Interval
The predictions that DRS gets from vROps are always for a certain period of time, starting from the current
time. This period is known as the “look ahead interval” for predictive DRS. This is by default 60 minutes starting from the current time, which means, by default the predictions will always be for the next one hour. So if there is any sudden spike that is going to happen in the next one hour, predictive DRS will detect it and will prepare the cluster to handle it.
Traditionally, DRS has always considered the compute resource (CPU and memory) utilizations of hosts and VMs for balancing load across hosts and placing VMs during power-on. This generally works well because in many cases, CPU and memory are the most important resources needed for good application performance.
However, since network availability is not considered in this approach, sometimes this results in placing or
migrating a VM to a host which is already network saturated. This might have some performance impact on the application if it happens to be network sensitive.
DRS is network-aware in vSphere 6.5, so it now considers the network utilization of host and network usage requirements of VMs during initial placement and load balancing. This makes DRS load balancing and initial placement of VMs more effective.
How It Works
During initial placement and load balancing, DRS first comes up with the list of best possible hosts to run a VM based on compute resources and then uses some heuristics to decide the final host based on VM and host network utilization’s. This makes sure the VM gets the network resources it needs along with the compute resources.
The goal of network-aware DRS in vSphere 6.5 is only to make sure the host has sufficient network resources available along with compute resources required by the VM. So, unlike regular DRS, which balances the CPU and memory load, network-aware DRS does not balance the network load in the cluster, which means it will not trigger a vMotion when there is network load imbalance.
Download a full DRS PERFORMANCE in VMware vSphere 6.5 Study Guide.
Microsoft SQL Server is one of the most widely deployed database platforms in the world, with many
organizations having dozens or even hundreds of instances deployed in their environments. The flexibility
of SQL Server, with its rich application capabilities combined with the low costs of x86 computing, has led
to a wide variety of SQL Server installations ranging from large data warehouses to small, highly
specialized departmental and application databases. The flexibility at the database layer translates
directly into application flexibility, giving end users more useful application features and ultimately
Application flexibility often comes at a cost to operations. As the number of applications in the enterprise
continues to grow, an increasing number of SQL Server installations are brought under lifecycle
management. Each application has its own set of requirements for the database layer, resulting in
multiple versions, patch levels, and maintenance processes. For this reason, many application owners
insist on having an SQL Server installation dedicated to an application. As application workloads vary
greatly, many SQL Server installations are allocated more hardware than they need, while others are
starved for compute resources.
This document provides best practice guidelines for designing Microsoft SQL Server on vSphere. The
recommendations are not specific to any particular set of hardware or to the size and scope of any
particular SQL Server implementation. The examples and considerations in this document provide
guidance only and do not represent strict design requirements, as varying application requirements would
result in many valid configuration possibilities.
vSphere Best Practices for SQL Server
A properly designed virtualized SQL Server using vSphere setup is crucial to the successful
implementation of enterprise applications. One main difference between designing for performance of
critical databases and designing for consolidation, which is the traditional practice when virtualizing, is
that when you design for performance you strive to reduce resource contention between virtual machines
as much as possible and even eliminate contention altogether. The following sections outline VMware
recommended practices for designing your vSphere environment to optimize for best performance.
3.1 Right Sizing
Right sizing is a term that is used when sizing virtual machines to contrast with sizing practices of physical servers. For example, a DBA determines that the number of CPUs required for a newly designed database server is eight CPUs. When deployed on a physical machine, typically the DBA will ask for more CPU power than the requirements at that point in time, sometimes even twice as much. The reason for this is usually that it is difficult for the DBA to add CPUs to this physical server after it has been deployed.
The general practice is to purchase the extra resources (CPU, disk, network, and memory) for the
physical server to accommodate for future growth requirements, sizing miscalculations, and any
unforeseen circumstances that can cause the database to require more resources in the future than
We will continue delivering this Fling on a regular basis even with the released supported version of the vSphere Client, so we hope that most of you will continue to use the Fling and update it weekly so that we can get your feedback about our direction.
You can find a summary of changes in all the clients available in vSphere 6.5 in this blog post – What’s new in vSphere 6.5: vCenter management clients. In the blog, you can also find a link to all the features that are not available in the GA version of the vSphere Client and link to frequently asked questions on all the vSphere Clients.
We would also like to thank all of you that have tried the Fling and have provided us feedback, we would not have achieved the progress we have without your engagement.
For partners who want to extend the HTML Client, this Fling also includes a new HTML SDK. Please see the HTML Client SDK Fling Overview.pdf and download the html-client-sdk.zip.
- Linux will show a warning on the login page about being unsupported, but the Fling should still work after login
- Bug: Unpatched versions of IE can fail to login and end up at URL “vmware-csd://csd”: This is a known vCenter 6.0 bug that we cannot patch with this appliance. The KB and fix are located here: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2109554
- Bug: If the client page is visited before the server has fully started, sometimes the strings do not load and you will see things like “summaryView.guestFullName” in views or in action menus. Refreshing the page may fix this, but restarting the appliance server with: “/etc/init.d/vsphere-client restart” should definitely fix it.
- This client is designed for managing vCenter. If you are looking for the Host Client you can find it as “ESXi Embedded Host Client” (https://labs.vmware.com/flings/esxi-embedded-host-client).
- The new visual theme is called “Clarity”, and is also still in development. We will be fixing UI bugs and subtly changing the look and feel to fit the new Clarity standard.
- The appliance requirements are recommended settings. It should still work at lower settings, but performance may be impacted.
- The root password on the appliance is ‘demova’ (without quotes). If you enter the wrong password too many times you will get locked out. The easiest way to fix this is to redeploy the appliance from the OVA.
- HTML Client SDK Fling Overview and Release Notes-5097204.pdf.
- Create a new certificate for a HTML5 client fling v3.pdf.
- H5 Client Deployment Instructions and Helpful Tips_v27.pdf.
VMware Fling page
Download h5ngcVA-126.96.36.199-5130996_OVF10.ova from vSphere HTML5 Web Client Fling page.
See previous HTML5 Web Client v2.5
|Configuration||Sphere 6.5||vSphere 6.0|
Virtual Machines Maximums
|RAM per VM||6128GB||4080GB|
|Virtual NVMe adapters per VM||4||N/A|
|Virtual NVMe targets per virtual SCSI adapter||15||N/A|
|Virtual NVMe targets per VM||60||N/A|
|Virtual RDMA Adapters per VM||1||N/A|
|Video memory per VM||2GB||512MB|
ESXi Host Maximums
|Logical CPUs per host||576||480|
|RAM per host||12TB||6TB *some exceptions|
|LUNs per server||512||256|
|Number of total paths on a server||2048||1024|
|FC LUNs per host||512||256|
|LUN ID||0 to 16383||0 to 1023|
|VMFS Volumes per host||512||256|
|FT virtual machines per cluster||128||98|
|Hosts per vCenter Server||2000||1000|
|Powered-on VMs per vCenter Server||25000||10000|
|Registered VMs per vCenter Server||35000||15000|
|Number of host per datacenter||2000||500|
|Maximum mixed vSphere Client (HTML5) + vSphere Web
Client simultaneous connections per VC
|60 (30 Flex, 30 maximum HTML5)||N/A|
|Maximum supported inventory for vSphere Client
|10,000 VMs, 1,000 Hosts||N/A|
|Host Profile Datastores||256||120|
|Host Profile Created||500||1200|
|Host Profile Attached||500||1000|
Platform Services Controller Maximums
|Maximum PSCs per vSphere Domain||10||8|
vCenter Server Extensions Maximums
|[VUM] VMware Tools upgrade per ESXi host||30||24|
|[VUM] Virtual machine hardware upgrade per host||30||24|
|[VUM] VMware Tools scan per VUM server||200||90|
|[VUM] VMware Tools upgrade per VUM server||200||75|
|[VUM] Virtual machine hardware scan per VUM server||200||90|
|[VUM] Virtual machine hardware upgrade per VUM server||200||75|
|[VUM] ESXi host scan per VUM server||232||75|
|[VUM] ESXi host patch remediation per VUM server||232||71|
|[VUM] ESXi host upgrade per VUM server||232||71|
Virtual SAN Maximums
|Virtual machines per cluster||6000||6400|
|Number of iSCSI LUNs per Cluster||1024||N/A|
|Number of iSCSI Targets per Cluster||128||N/A|
|Number of iSCSI LUNs per Target||256||N/A|
|Max iSCSI LUN size||62TB||N/A|
|Number of iSCSI sessions per Node||1024||N/A|
|iSCSI IO queue depth per Node||4096||N/A|
|Number of outstanding writes per iSCSI LUN||128||N/A|
|Number of outstanding IOs per iSCSI LUN||256||N/A|
|Number of initiators who register PR key for a iSCSI LUN||64||N/A|
Storage Policy Maximums
|Maximum number of VM storage policies||1024||Not Published|
|Maximum number of VASA providers||1024||Not Published|
|Maximum number of rule sets in VM storage
|Maximum capabilities in VM storage policy
|Maximum vSphere tags in virtual machine storage policy||128||Not Published|