everRun Release 7.3.1.1 Release Notes

These release notes are for everRun release 7.3.1.1 (updated at 6:05 PM on 4/9/2018). See the following sections:

• Important Considerations
• Known Issues
• New Features, Enhancements, and Bug Fixes
• Getting Help

Important Considerations

Upgrading From Previous Releases of everRun

You can upgrade from any prior everRun release (listed on the Downloads page at http://www.stratus.com/go/support/everrun) to everRun release 7.3.x with no VM downtime by following the instructions in Upgrading everRun Software.

If you are upgrading from a release not listed on the everRun Support page (for example, a beta release) you must perform a complete system re-install.

Upgrading From everRun 7.1.x Using a DVD

If you plan to upgrade a 7.1.x everRun system using a 7.3.x DVD, before you begin the upgrade contact your authorized Stratus service representative for important information about how to perform this procedure.

Upgrading an everRun 7.2.x DR Environment

If you are using the DR feature on everRun 7.2.x systems with One View 1.0.x and you want to upgrade to everRun 7.3.x and One View 2.0.0.0, see the release notes in the Stratus One View Console and everRun Disaster Recovery User's Guide for special upgrade instructions.

A DR-Protected VM Cannot Be Deleted

A DR-protected VM cannot be deleted. After shutting down a DR-protected VM, the Remove button will not appear. To delete such a VM, use the One View Console to remove DR protection (unprotect) from that VM. After DR protection is removed, the Remove button will be enabled and you can use the everRun Availability Console to shut down and remove the VM.

Updating Guest VM Software After Installing a VM

After installing a VM, check to see if updates to the guest OS are available. If updates for your OS are available, install them. However, use only the virtIO drivers provided with the everRun software. Do not upgrade virtIO drivers.

Do Not Update CentOS Host OS Directly From CentOS

Do not update the CentOS host OS software directly from CentOS. Use only the CentOS release that is installed along with everRun software.

Optimizing Performance of A-Link Networks

Stratus recommends that you enable jumbo frames on A-Link networks by setting their Ethernet frame MTU size to 9000 bytes (by default, they are set to 1500 bytes). Doing so improves VM performance and reduces host processing overhead.

The A-Link networks must:

• Be a single Ethernet cable, point-to-point connection, or
• Have intermediate components (for example, switches) that are all fully capable of passing jumbo frame traffic

Use AVCLI commands to enable jumbo frames. AVCLI is installed on the host system along with everRun software. You can run AVCLI by logging onto the host through a remote console using the system IP address. Alternatively, you can install AVCLI on a remote management computer. For information about how to install AVCLI on a remote computer, see AVCLI Command Overview.

To set A-Links to use jumbo frames:

1. From a remote console, issue the network-info command to determine the name(s) of the A-Link network(s). In the command's output, look for the names of networks that have role = A-Link . See network-info for an example.

2. Issue the network-change-mtu command to change the MTU size to the maximum value of 9000 bytes. The change takes effect immediately. The following example changes the sync_2003 and sync_2004 A-Link networks to use jumbo frames.

   avcli network-change-mtu sync_2003 sync_2004 9000
3. Issue the network-info command to verify that the A-Links now have MTU values of 9000.

Migrating a PM or VM to an everRun System

Migrating Windows 2012 R2 or Windows 8.x PMs or VMs from a non-everRun system to an everRun system is not supported. For a list of PM and VM operating systems that you can migrate, see Migrating a Physical Machine or Virtual Machine to an everRun 7.x System.

Status of RAID Physical Disks Is Not Monitored

everRun software does not monitor the state of physical disks in a RAID set. You must use the tools supplied by the RAID controller vendor to monitor the health and status of individual physical disks in a RAID set.

Other Important everRun Considerations

For important considerations about everRun systems, see Important Physical Machine and Virtual Machine Considerations.

Known Issues

Windows 2008 (pre-R2) Guests May Crash

Windows Server 2008 (pre-R2) VMs are at risk of crashing and can exhibit the following symptoms:

• Bug Check 0x19: BAD_POOL_HEADER (see http://msdn.microsoft.com/en-us/library/windows/hardware/ff557389(v=vs.85).aspx for more information).
• Bug Check 0x3B: SYSTEM_SERVICE_EXCEPTION (see http://msdn.microsoft.com/en-us/library/windows/hardware/ff558949(v=vs.85).aspx for more information).

Under heavy disk loads, such as during a back up, Windows Server 2008 x64 pre-R2 (NT 6.0, Vista kernel) crashes with the symptoms listed above.

Guest VMs running other Windows versions, including Windows Server 2008 R2 and Linux do not exhibit these symptoms.

Do not install or import Windows Server 2008 pre-R2 guests on everRun systems. Instead install one of the following supported Windows types which do not exhibit these symptoms:

• Windows Server 2008 R2 (NT6.1, Windows 7 kernel) x64
• Windows Server 2012 (NT6.2, Windows 8 kernel) x64
• Windows Server 2012 R2 (NT6.3, Windows 8.1 kernel)

If you already use Windows Server pre-R2 guests, monitor them for the symptoms described above. Install a replacement VM of one of the Windows types mentioned above for any VM that must run reliably. If crashes do not impact you, it is likely that crashes will occur as load on your VM increases.

e-Alert Message Security

When Connect using TLS is selected on the e-Alerts Preferences page, e-Alerts are not encrypted and are transmitted on the non-secure port 25.

UEFI Boot Mode Prevents everRun Installation

Before installing everRun software on a physical machine, verify that the system's BIOS is not set to UEFI boot mode. If it is, change the BIOS boot mode setting to legacy boot mode.

Node IP Address Change Requires Reboot

After changing a node's IP address on the IP Configuration Preferences page in the everRun Availability Console, the new address does not take effect until after you reboot that node.

VCD Creation, VM Import, and VM Restore May Not Work Using Chrome Browser

VCD creation, VM import, and VM restore operations make use of Java Applets running in a browser. Starting with version 42, the Chrome browser does not support the NPAPI plug-in architecture upon which Java Applets are based. When performing any of these operations, use a browser that supports NPAPI.

Number of VCPUs May Not Be Displayed Correctly

In rare instances, the number of VCPUs displayed on the System page is incorrect. You can use the CentOS lscpu command to verify the number of VCPUs.

The Activate Foreign Operation On an External Disk May Fail

If clicking Activate Foreign for an iSCSI or Fibre Channel external logical disk fails to activate the disk, perform the following steps to activate it:

1. Using your storage system's software, unmap the disk from the everRun system's host adapters.
2. Delete the logical disk from the storage array.
3. Perform a rescan on the everRun system's host adapters to remove the deleted logical disk.
4. Using your storage system's software, re-create a new logical disk and map it to the everRun system's host adapters.
5. Perform another rescan on the everRun system's host adapters to recognize the new logical disk.

6. The new logical disk should show up on the Storage tab on the Physical Machines page as not foreign and you can move it to a storage group.

See Adding or Removing External Storage LUNs for more information about deleting logical disks and performing rescan operations.

Alert Status For a Failed Volume May Not Be Displayed Correctly

An alert indicating that a volume has failed on a specific node may have a Normal status indicator (green check mark) on the Alerts page even though the volume remains in a failed condition. In such a case, the Repair Storage button will still be visible. Click it to move the volume to another logical disk.

e-Alerts for Missing External Storage May Be Incomplete

If an external storage disk is unavailable and alerts are generated, the resulting e-Alerts may be missing a substantial amount of text.

Volumes May Remain After VM Is Removed

Occasionally after removing a virtual machine, one of the volumes that was selected in the Remove Virtual Machine dialog box, may remain on the system. If this occurs, you can explicitly delete the volume on the Volumes page.

everRun May Not Detect Reconnection of Management Network Cable

If you boot a PM while its management network (ibiz0) cable is disconnected and then you reconnect the cable after the PM is running, everRun may not detect that the cable has been reconnected. If this occurs you can 1) reboot the physical machine (with the cable connected) or 2) issue the following commands from the other unaffected PM in the system.

VCD Replication Can Fail During Heavy Disk Activity

During heavy disk activity, VCD replication can fail and the VCD can go into a Warning state. If this occurs, remove the VCD and create it again.

7.3 Upgrade Failure Alert While PM Is Booting May Be Inaccurate

If you receive an alert, while a PM is still booting, indicating that the upgrade to everRun7.3 has failed, the failure may not have actually occurred. Wait until the PM finishes booting and then check the status of the alert on the Alerts page. If its status changes to Normal (green check mark) the upgrade was successful.

Volume Names In Alerts

When volume names that include hyphen characters (-) appear in alerts, the hyphens (-) in their names may be omitted.

VM Name and Description Do Not Take Effect When Reprovisioning

The VM name and description entered into the Reprovision Virtual Machine wizard do not take effect. After the reprovision operation completes, you can name the VM (see Renaming a Virtual Machine for details) and you can enter its description on the Description tab on the Virtual Machines page.

VM Load Balance After Upgrade to everRun 7.3

After upgrading to everRun 7.3, a message in the masthead may appear stating that the VMs are not load-balanced when in fact they are. No action is necessary.

Alternatively, you may notice that after an upgrade the VMs are actually not load-balanced. If this occurs you can load balance the VMs. See Load Balancing for details.

New Size of Windows VM Volume Not Displayed

If you perform an expand volume operation on a Windows VM volume when one system node is down, the new volume size is not visible in the VM until both PMs are synchronized and running.

Information About PCI Devices May Not Be Displayed

Depending on a system's hardware configuration, information about PCI devices (and the storage attached to them) may not be displayed in the everRun Availability Console.

State of External Storage Not Reflected in Node's State

A PM's (node) state is not affected by the state of non-system dependent volumes (any volumes other than root, swap and shared.fs). As a result, external storage can be in a failed state and the node's state will not reflect that. To see the state of external storage, view the state of the VMs and volumes that are on external storage.

Simplex PM Does Not Reboot After an everRunUpgrade

When upgrading a simplex system (single PM) from everRunRelease 7.2.0.0, the upgrade process completes but the system does not reboot. To reboot a simplex system after it's upgrade completes, click System in the left-hand navigation panel then click Reboot.

VMs May Not Boot If One Node is Removed From System

If you remove one PM from the system (by clicking Work On and then clicking Remove), do not place the remaining PM into maintenance and then bring it back into service by just using the Finalize button. Doing so will prevent all VMs from being able to boot. If you must place the remaining PM into maintenance mode, reboot it before you return it to service. See Rebooting a Physical Machine for details.

Uploading an Upgrade Kit Will Fail If The User Session Times Out

Uploading an upgrade kit will fail if the user session on the everRun Availability Console times out during the upload process. This might occur, for example, when upgrading a DR PM through the primary PM's site over a low-bandwidth or heavily congested DR link. If the upload process is taking a long time, you can avoid the problem by performing an action in the everRun Availability Console (like clicking to view a different page) before the session times out. Alternatively you can perform the upload locally at the site of the system being upgraded.

A VM Exported With Only Some of its Snapshotted Volumes Cannot be Imported

During a VM export operation, be sure to select all of the VMs volumes. Doing so exports a VM that can then be imported at a later time.

Removing User or DR Snapshots Temporarily Prevents Some VM and DR Operations

When either a user or the Disaster Recovery (DR) software removes a snapshot on an everRun system, the system must coalesce the snapshot by merging it with the next oldest snapshot. While the system is coalescing snapshots:

• A user cannot create a new snapshot in the everRun Availability Console. If you try, an error indicates that the system is busy.
• The DR software cannot create DR snapshots of the primary VM. If DR snapshots are delayed long enough, DR protection may temporarily fall below your snapshot retention and recovery point objective (RPO) thresholds until DR snapshots resume.
• A user cannot start the VM associated with the snapshot(s) if the VM is currently stopped. The Start button is temporarily unavailable on the Virtual Machines page of the everRun Availability Console.
• A user cannot enable or resume DR protection for a VM. An alert on the Alerts page of the everRun Availability Console may indicate that there is not enough storage space for the snapshot, because the coalescing snapshots continue to occupy space in the volume container(s) until they are finally removed.

Avoid removing snapshots if you have an immediate need to perform any of these operations. After you remove a snapshot, wait at least 10-15 minutes before attempting any of these operations, or retry the operation if needed. You may need to wait much longer depending on the size of your volumes, the amount of VM activity, and the number of snapshots you remove.

For DR-protected VMs, if DR snapshot replication appears to stop or fall below your thresholds, check the Alerts page of the everRun Availability Console for more information.

Coalescing Snapshots Can Affect RPO

When snapshots are coalescing, new snapshots cannot be taken until coalescing completes. If the RPO is set to be near or lower than the typical coalesce time for a VM's activity level, the VM will periodically fall out of RPO. This is unlikely to happen on systems with moderate work load and RPOs in the 1 - 6 hour range but should this occur, it is necessary to increase the RPO time to avoid falling out of RPO.

Moving an everRun System to a Different Subnet

Use the following procedure to prepare the management IP configuration of an everRun system to work on a different subnet (for example when shipping to another location or when reconfiguring network subnets).

1. In the everRun Availability Console, click Physical Machines in the left-hand navigation panel, and note which node is the primary node.
2. Shut down all running VMs.
3. Connect a VGA console directly to the primary node.
4. On the VGA console, press Enter. The console displays several addresses. Note the PM's link-local IPv6 address (it is the one that begins with fe80::).
5. In the management computer's browser, enter the URL of the link-local IPv6 address in the form http://[ipv6-link-local-address]. Be sure to include the address within brackets, for example: http://[fe80::21c:23ff:fedd:30ed].
6. Place both nodes into maintenance mode. Log on to the everRun Availability Console and in the left-hand navigation panel, click Physical Machines.
   1. Select the secondary node (the one not marked primary) and click Work On.
   2. Select the primary node and click Work On.
7. Click Preferences in the left-hand navigation panel, and click IP Configuration.
   1. Change the IP configuration settings to match the addresses for the new subnet to which the system will be moved.
   2. Click Save.
8. Click System in the left-hand navigation panel, then click Shutdown.
9. Move the everRun system to the new location and/ or connect it to the new subnet.
10. Power on both PMs.
11. On a management computer, connect to the everRun Availability Console using the IPv4 management address you specified in step 7.
12. Remove both nodes from maintenance mode. In the left-hand navigation panel, click Physical Machines.
    1. Select one node and click Finalize.
    2. Select the other node and click Finalize.

Under Certain High Workloads Windows VM Snapshots May Not Be Application-Consistent

In some high workload situations, the Windows QEMU guest agent may become unresponsive, preventing snapshots of Windows VMs from being application-consistent (instead they are only crash consistent as indicated on the Summary tab of the Snapshots page). If this occurs, application-consistent snapshots will no longer be possible until the Windows QEMU quest agent is restarted. Perform the following procedure to restore application-consistent snapshot capability.

1. In the Windows Task Manager stop the QEMU Guest Agent (qemu-ga.exe) process.

2. In the Windows Services user interface, start the QEMU Guest Agent service.

Specifying a Log File While Installing the Windows QEMU Guest Agent May Cause VM Timeouts

Do not specify a log file while installing qemu-ga.exe. Doing so may cause VSS timeouts when snapshots are being taken.

Replacing Physical Machines, Motherboards, NICs, or RAID Controllers

In the event of a hardware failure requiring replacement of an entire PM, everRun supports PM replacement while VMs continue to run on the other PM, thereby avoiding any guest downtime. The procedure for doing this is documented in Replacing Physical Machines, Motherboards, NICs, or RAID Controllers. After completing this procedure, perform the following steps to avoid any potential subsequent problems.

1. The preferred PM setting can get corrupted after replacing a PM. It is not possible to tell which VMs may be impacted by this, so you must perform steps a. and b. below for each VM.
   1. In the everRun Availability Console, change the preferred PM setting to the node that the VM is not currently on. See Load Balancing and Selecting a Preferred PM for a Virtual Machine for information about how to do this.
   2. Then change the preferred PM of each VM back to the desired setting.
2. If, after performing the previous step, an alert appears saying that a rebalance must be performed, on the everRun Availability Console masthead, click Rebalance ().

Unsupported Network Adapter Card and Chip

Due to the issue outlined at http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=migr-5093183, everRun does not support the following network adapter card and chip:

• The Broadcom NetXtreme II Dual Port 10GBase-T Network Adapter, IBM part number 49Y7910
• Any other NICs that use the Broadcom BCM57712 Ethernet hardware chip

Do Not Use the ifdown Command

Do not issue the ifdown command from an everRun physical machine's host OS to temporarily bring down a VM's business (ibizx) network connection. Doing so will disconnect the physical interface from its bridge and cause the VM to become unreachable over the network. Instead, use the ifconfig down command.

New Features, Enhancements, and Bug Fixes

Major new features, enhancements, and/or bug fixes are listed below under the release in which they became available.

Fixed in everRun Release 7.3.1.1

• bz-31557: Disk scrubber process shows evidence of leaks in versions 7.3.0.0 and later releases.

• bz-31556: HCT fix in support of Microsoft virtualization certification.

Fixed in everRun Release 7.3.1.0

• Updated the content of localized user documentation
• bz-29844/bz-29846: Windows 2003 and Windows 2008 pre-R2 guests may experience ten second network blackouts.
• bz-30036: Five second network blackout can occur during upgrade from 7.1.x or 7.2.x.
• bz-30500: For some IBM systems, the number of VCPUs was being incorrectly calculated.
• bz-30548: On occasion, ibiz0 would not be restored following the recovery of a NIC hang.
• bz-30571: The CentOS 6.6 kernel is being updated to a later rev in order to get bug fixes for issues that affect newer platforms.
• bz-30644: For certain Intel white boxes, everRun installs would fail.
• bz-30867: Upgrading from 7.2.3 to 7.3.x caused the everRun portal to have incorrect formatting.
• bz-30878: On rare occasions, creating a diagnostics file could cause a guest bugcheck.
• bz-30909: The everRun EULA has been updated.
• bz-31073: Behavior change: on a fresh install, only the logical disk which has been installed to is placed in Initial Storage Group. All other logical disks must be manually placed into storage groups prior to use.
• bz-31112: KVM updates are being accepted that are associated with the CentOS 6.6 kernel update.

New in everRun Release 7.3.0.0

• Support for multiple storage groups
• Support for external iSCSI and Fibre Channel storage

Getting Help

If you have a technical question about everRun software, you can find the latest documentation at the Downloads page at http://www.stratus.com/go/support/everrun.

If you are unable to resolve your questions with the online documentation, and the system is covered by a service agreement, please contact everRun Customer Support or your authorized Stratus service representative. For information, see the everRun Support page at http://www.stratus.com/services-support/customer-support.