VMware ESXi Hypervisor Installing and Setting Up

From this article, you will learn about VMware ESXi.

General information

Hardware

• VMware vSphere Hypervisor 5.x and 6.0 (ESXi) use a filesystem called vmfs5 (formerly vmfs3), which uses GPT and supports drive sizes of up to 64 TiB (formerly 2 TiB).
• Software RAID is not supported by ESXi.
• The free version of ESXi (starting with 5.5) supports a maximum of 4 TB RAM (formerly 32 GB RAM).

Network configuration

• VMware vSphere Hypervisor is an operating system designed for pure virtualization and supports neither NAT nor routing. Therefore, you can only use a true bridged setup.
• To use a subnet, you need to set up an additional single IP as a router VM.

Installation

• The installation and configuration of ESXi takes about 20-30 minutes, even without any prior knowledge.

License

• After installation, the system has a test license which expires after 60 days. If you want to use the OS for the long term, you need to get a separate (free) license. You can do this by registering on the VMware website:
  • License for ESXi 5.x
  • License for ESXi 6.0
• You can enter the license via the vSphere Client. Go to the Configuration tab, the Software section, and then choose Licensed functions in the menu. After selecting this, you can click on the Edit button to the right and then enter the license key.

Specific information

Installed hardware

The Dell PowerEdge models DX150, DX151, DX290, DX291 and DX141 are certified and compatible with all versions from 5.0 onwards. DX152 and DX292 are compatible from 6.0 onwards. DX180 and DX181 are compatible from 6.5 onwards.

All other models are not certified by VMware, but in most cases, you can run them with VMware vSphere/ESXi.

Compatibility

We do not officially test or guarantee compatibility with our servers. Please check yourself if they’re compatible or not.

NOTES:

• The Realtek driver present in ESXi 5.0 to 5.1 Update 2 was just a tech demo that has not been updated. Therefore, to make sure your system operates smoothly, you should use a server with an Intel NIC (PX60, PX91, DX151 etc.).
• To install vSphere 5.5 or newer on the EX40, EX60 and some EX41/EX51 models, you need to use a custom installed ISO that includes drivers for the Realtek NIC (http://www.bussink.ch/?p=1228). Or you can install an additional compatible NIC.
• It is not possible to install ESXi on the older DS or X servers.
• During installtion on a server with multiple identical drives, ESXi can potentially show the drives in a different order than the BIOS does. If the screen is blank with only a blinking cursor after you are done the installation and have rebooted, try booting from the other drive.

Network configuration

• For connectivity between multiple ESXi servers within the same subnet, you need set up host routes via gateway because of network security restrictions.

host A
esxcfg-route -a <IP Host B> 255.255.255.255 <Gateway IP>

host B
esxcfg-route -a <IP Host A> 255.255.255.255 <Gateway IP>

Single IP addresses

• IP addresses are, by default, statically mapped to the MAC address of the host. It is possible, however, to get separate MAC addresses for the additional single IPs via Robot. You can then configure these can for the virtual machines. To get these MAC addresses assigned, log into Robot, choose “Server” from the menu on the left, select the correct server, and then click on the “IPs” tab. Then click on the icon next to the single IP address to get a MAC address.

Subnets

• To use a subnet (IPv4 as well as IPv6) in ESXi, you need to have at least one additional single IP set up as a router VM, since ESXi itself cannot route. When ordering a subnet, please make sure include a note that you want to use it for ESXi and it should be routed on the additional single IP.

IMPORTANT Since IPv6 subnets are routed to link-local addresses (MAC-based), it is only possible to use IPv6 in a limited way (meaning on a single VM).

IPv4

The confirmation email of the subnet contains (for example) the following information:

Below you will find the IP subnet added to your server 192.168.13.156.

Subnet: 192.168.182.16 /28
Mask: 255.255.255.240
Broadcast: 192.168.182.31

Usable IP addresses:
192.168.182.17 to 192.168.182.30

You do NOT get a separate MAC for each IP from a subnet.

IPv6

All servers come with a /64 IPv6 subnet. To see which IPv6 subnet your server has, please check the IPs tab of the server in the client’s area.

The IPv6 subnet is routed to the default link-local address (which is derived from the MAC address) of the main IP. You can switch the routing of the IPv6 subnet to the link-local address of the virtual MAC (in other words, the additional single IP). You can do this in the client’s area using the same symbol which is located next to additional single IPs. It’s the same symbol you use to request virtual MAC addresses. The host system, meaning the ESXi itself, receives no IPv6 address. This is neither necessary nor possible because ESXi cannot work with a fe80::1 gateway.

In order to use these IP addresses in virtual machines, you need a router VM supplemented by an additional virtual NIC from the new subnet. The subnet itself requires a new vSwitch in ESXi which all VMs in the subnet will be connected to.

Notes

The network card type for the router VM should not be VMXNET2 or VMXNET3; otherwise, the TCP performance can be very bad. As a workaround, you can disable LRO in the VM with disable_lro=1. You can find more information on this bug here: http://www.vmware.com/support/vsphere4/doc/vsp_esxi41_vc41_rel_notes.html

After an upgrade to VMware ESXi 5, this issue may appear again. You can address this issue by disabling Large Receive Offload (LRO) on the ESXi host:

• Log into the ESXi host with the vSphere Client.
• Select the host -> Configuration -> Software:Advanced Settings.
• Select Net and scroll down, slightly more than half way.
• Set the following parameters from 1 to 0:

Net.VmxnetSwLROSL
Net.Vmxnet3SwLRO
Net.Vmxnet3HwLRO
Net.Vmxnet2SwLRO
Net.Vmxnet2HwLRO

Reboot the ESXi host to activate these changes.

If you experience connection problems in systems with Realtek network cards, then you can solve this under certain circumstances by deactivating offloading and activating polling. However, this also reduces the performance.

• checksum offload: deactivated
• segmentation offload: deactivated
• large receive offload: deactivated
• device polling: enabled

Preparations in the vSphere client

1. Create a vSwitch (in the example, the name “subnetz” is used)

2. Add a second NIC to the router VM. Connected network: subnetz (the previously created vSwitch)

3. The NIC of the virtual machine in the subnet. Connected network: subnetz

The networking overview should show the following:

Configuration of the Router VM

• Example of /etc/network/interfaces on the router VM

# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
# WAN-NIC im VMnetwork
auto eth0
iface eth0 inet dhcp
# for the IPv6 subnet the configuration is analog to other virtualisations
iface eth0 inet6 static
 address 2a01:4f8:61:20e1::2
 netmask 128
 gateway fe80::1
# LAN NIC in Subnet
auto eth1
iface eth1 inet static
 address     192.168.182.30
 netmask     255.255.255.240
# The prefix/netmask can/must be changed according to the amount of network
# segments
iface eth1 inet6 static
 address    2a01:4f8:61:20e1::2
 netmask    64

• Example of /etc/network/interfaces of a Linux VM in the subnet

# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
 address 192.168.182.17
 netmask 255.255.255.240
 gateway 192.168.182.30
iface eth0 inet6 static
 address    2a01:4f8:61:20e1::4
 netmask    64
 gateway    2a01:4f8:61:20e1::2

The router VM is now connected to both networks, and you can use it as a gateway for virtual machines in the subnet. Don’t forget to activate IP forwarding in the kernel:

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding

To make this persistent across reboots, add the following line to /etc/sysctl.conf

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

The virtual machines should now be accessible (via SSH for example) via their assigned IPs.

Installation Guide

• Choose the Rescue System as the OS for the server you order.
• If you would like RAID, you can add a 4-Port RAID controller since ESXi doesn’t support software RAID.
• Once the server is online (you will get an email from us informing you of this), you can order a KVM Console and use that to virtually mount an ISO file of the version of ESXi you want to install.

Then, you should see the following screen:

• After a reboot, log in using the password that you entered during the installation. This is the root password for SSH as well as the password for the VMware vSphere Client (requires Windows). You can then download this separately via a browser.

• After you’re finished the installation, you can order up to three additional single IP addresses via Robot. You can get a MAC address for your additional single IP on Robot under the IPs tab. You will see a small button next to the IP address. Clicking on that button will give the IP address a virtual MAC.
• Configure the MAC addresses of the virtual servers with the corresponding IP addresses with vSphere. Once you have done this, even DHCP will work via our network!

For additional information regarding ESXi and its usage, please refer to the official website

Manual installation of updates

You can only do the installation of updates in the free version via the console or via VMware Go. An update can be several hundred megabytes, which can take a long time with a standard DSL connection, which is why the following guide can help. Follow this guide at your own risk. There is no warranty or guarantee that it will work!

Before you start, you need two things. First, you need SSH activated access. And second, the system needs to be in maintenance mode. You can activate this using:

vim-cmd hostsvc/maintenance_mode_enter

Updating vSphere 5.0 to 5.1

First of all, you need to download the VMware-ESXi-5.1.0-799733-depot.zip update from the VMWare Updates Page and save it on the ESXi host.

Once all the VMs have been shut down and the system has entered the maintenance mode via vim-cmd hostsvc/maintenance_mode_enter, you can use one of two ways to install the update. The following command refreshes the system and removes all the packages that are not included in the update. This is equivalent to a new installation.

esxcli software profile install -d /vmfs/volumes/datastore1/VMware-ESXi-5.1.0-799733-depot.zip -p ESXi-5.1.0-799733-standard

Or, you can only update the packages containted in the update to their new versions, leaving all other packages intact.

esxcli software profile update -d /vmfs/volumes/datastore1/VMware-ESXi-5.1.0-799733-depot.zip -p ESXi-5.1.0-799733-standard

Finally, you need to reboot the system. When you turn the VMs on for the first time after the reboot, you might see a message noting that the VM was copied or moved. This happens because the UUIDs get changed during the update. You can safely select “VM was moved” in this situation. See also: VMware Help Page

Installation of patches

After you have transferred the patches onto the system, you can install them. It is important that you enter the full path name, eg.:

esxcli software vib install --depot="/vmfs/volumes/datastore1/patches/ESXi510-201210001.zip"
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
[...]

After a reboot, you need to exit from the maintenance mode:

vim-cmd hostsvc/maintenance_mode_exit

Monitoring RAID controller

3ware Controller

For the 3ware controller, there is both a CIM Provider as well as a CLI. You can use the 64-bit CLI for Linux from version 9.5.2 onwards.

Note: 3ware controllers are only supported by ESXi 5.0 via an external driver.

Adaptec Controller

For Adaptec controllers, you need to manually install the CIM Provider and the CLI (arcconf). You also need to have an up-to-date version of the driver. You can find an installation guide on the Adaptec website (PDF).

• RAID driver Version 5.2.1.29800
• Remote Arcconf
• Adaptec CIM Providers

You can start monitoring by installing remote ARCCONF via a Windows/Linux system.

$ arcconf GETCONFIG 1 AD

LSI Controller

LSI provides a CIM/SMIS provider. After you’ve finished the installation, the hardware monitoring page in the vSphere client displays the status of the RAID. However, you can only get an active alarm in the paid version and when running vCenter.

Or you can install the command line tool MegaCLI, which you can also use to manage the RAID controller. You can use a script to automate the display of status information. But you need to run this script and notifications from another server.

Parallel operation of onboard controller/hardware RAID

During the installation, ESXi only sees one type of storage, so either the onboard SATA controller or an additional RAID controller. If drives are connected to both, then the hardware controller is prioritized, and the drives connected to the onboard controller are invisible. By manually loading the appropriate kernel module, you can still use these drives.

/sbin/vmkload_mod ahci

To have this module loaded automatically during start, you need to add the line above to /etc/rc.local and /sbin/auto-backup.sh.

Hardware change

Change MAC address

In the event of a hardware (ex)change, especially the motherboard, it should be noted that the ESXi host retains its original MAC address. This leads to issues. The switch will not automatically forward the correct new main IP to the server, as the MAC address that is being broadcast is incorrect. You need to reset the MAC address via the ESXi shell. There are several approches for doing this, listed in the following VMWare Knowledge Base Article. The most elegant solution is when the ESXi host automatically recognizes the new MAC address when changing platforms and uses that. You can use the following command for that:

esxcfg-advcfg -s 1 /Net/FollowHardwareMac

Either perform this command before the platform change or, if the change has already happened, there are two options:

• Order a KVM Console console and enable the ESXi shell, and then press Alt + F1 to switch to the console and enter the command. Afterwards, pressing Alt + F2 will bring you back to the GUI.
• Temporarily teach the switch the new MAC address by booting into the Rescue System and then back into the ESXi host. As a result, the ESXi host will now be reachable again via the main IP, but only for a limited amount of time. The length of this time depends on how long the switch takes to delete the ARP cache entry for this MAC address. Normally there is enough time to log in via SSH and execute the command, assuming that you have enabled SSH access. However, even this is configurable, as connecting via the ESXi client would be possible again.

With either option, you need to do a restart afterwards. You can initiate this via the console:

reboot

After a restart, the MAC address should be set correctly and you can verify this in the ESXi shell with the following command:

esxcfg-vmknic -l

The new MAC address should show up next to the main IP.