Thursday, February 22, 2018

Voicemails left with Skype for Business Server 2015 on Exchange Server 2016 does not get delivered to user’s inbox

Problem

You’ve just completed integrating Skype for Business Server 2015 with Exchange Server 2016 Unified Messaging to provide voicemail services but noticed that voicemails left for users are not being delivered and the following errors are being logged on the Exchange 2016 server:

Log Name: Application

Source: MSExchange Unified Messaging

Event ID: 1423

Level: Error

image

The Microsoft Exchange Unified Messaging service on the Mailbox server encountered an error while trying to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\c7d3e3ee-0212-4d27-99bd-f1960eca0698.txt". Error details: "Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. ---> Microsoft.Exchange.Net.ExSmtpClient.UnexpectedSmtpServerResponseException: Unexpected SMTP server response. Expected: 220, actual: 421, whole response: 421 4.3.2 Service not available

at Microsoft.Exchange.Net.ExSmtpClient.SmtpTalk.CheckResponse(ServerResponseInfo response, Int32 expectedCode)

at Microsoft.Exchange.Net.ExSmtpClient.SmtpTalk.Connect(String server, Int32 port, Boolean isSslPort)

at Microsoft.Exchange.Net.ExSmtpClient.SmtpClient.Submit()

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, InternalExchangeServer smtpServer)

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId)

--- End of inner exception stack trace ---

Server stack trace:

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.HandleTransientSmtpFailure(Exception e, InternalExchangeServer smtpServer, String recipientAddress)

at Microsoft.Exchange.UM.UMCore.SmtpSubmissionHelper.SubmitMessage(MessageItem message, String senderAddress, Guid senderOrgGuid, String recipientAddress, OutboundConversionOptions submissionConversionOptions, String requestId)

at Microsoft.Exchange.UM.UMCore.SmtpSubmitStage.InternalDoSynchronousWork()

at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)

at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [0]:

at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)

at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)

at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.SynchronousWorkDelegate.EndInvoke(IAsyncResult result)

at Microsoft.Exchange.UM.UMCore.SynchronousPipelineStageBase.EndSynchronousWork(IAsyncResult r)"

image

Log Name: Application

Source: MSExchange Unified Messaging

Event ID: 1446

Level: Error

image

The Microsoft Exchange Unified Messaging service on the Mailbox server failed to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\1fbf0f79-adeb-4b6c-a7e2-792ec27a5351.txt" within "11" minutes. The server will continue to process and deliver the message, but the "MSExchangeUMAvailability: % of Messages Successfully Processed Over the Last Hour" performance counter will be decreased.

image

Solution

Similar to one of my previous posts from 7 years ago:

Voicemails left on Exchange UM does not get delivered to inbox – Event ID 1082 and 1035 logged
http://terenceluk.blogspot.com/2010/07/voicemails-left-on-exchange-um-does-not.html

… the issue was caused by the receiving connector on the Exchange server which hosts the Unified Messaging role (note that Exchange 2016 no longer splits roles into other servers).  The following snippet from the event ID 1423 error logged displays what response the Exchange server is receiving when an attempt to deliver the voice is made:

The Microsoft Exchange Unified Messaging service on the Mailbox server encountered an error while trying to process the message with header file "E:\Program Files\Microsoft\Exchange Server\V15\UnifiedMessaging\voicemail\c7d3e3ee-0212-4d27-99bd-f1960eca0698.txt". Error details: "Microsoft.Exchange.UM.UMCore.SmtpSubmissionException: Submission to the Hub Transport server failed. The operation will be retried. ---> Microsoft.Exchange.Net.ExSmtpClient.UnexpectedSmtpServerResponseException: Unexpected SMTP server response. Expected: 220, actual: 421, whole response: 421 4.3.2 Service not available

You can verify this by logging onto one of the Exchange servers and initiate a telnet to port 25 to another as shown here:

421 4.3.2 Service not available

Connection to host lost.

C:\>

image

If the environment you’re working in has many receive connectors configured:

image

… my suggestion is to not to attempt figuring out which routing connector may be receiving the request and rejecting the connection but rather create new ones specifically for UM with the the scope configured with the Exchange server IP address and the following security configuration:

image

You should see the queued files stored in the folder:

image

… begin to clear out when the Exchange server begins accepting the voicemail delivery connections.

Thursday, February 15, 2018

VMware Horizon View 7.4.0 virtual desktops does not automatically power on after upgrade

I recently upgraded a client’s VMware Horizon View 7.0.2 infrastructure to the latest 7.4.0 version Build 7400497:

image

image

… and received reports the next day that virtual desktops were not being started up when the user had shut them down even though the Remote Machine Power Policy was configured as Ensure machines are always powered on:

image

After reviewing the event logs and performing various troubleshooting steps without any luck, I gave the recommendation VMware support always suggested:

Shut down all View servers then power them on one after another to allow the database to synchronize properly

… a try and was able to correct the issue.  Hope this post will help anyone who may encounter this problem.  Please refer to the following KB for a more detailed outline of the steps for restarting the servers:

Restart order of the View environment to clear ADLDS (ADAM) synchronization in Horizon View (2068381)
https://kb.vmware.com/s/article/2068381

Saturday, February 3, 2018

Upgrading VMware Horizon View from 7.0.2 to 7.4.0 causes VMware Horizon Composers to display a Red status in the View Administrator with "The service is not working properly" and SSL as "Unknown"

Problem

You have just successfully upgraded your VMware Horizon View 7.0.2 environment to 7.4.0 but noticed that the System Health of the View Composer Servers now displays a Red status with The service is not working properly and SSL as Unknown with no option of accepting the self signed certificate that the Composer service is using as you would normally expect to see.

The View 7.4.0 environment is currently using an older vSphere 5.5 environment and you have confirmed that TLS 1.0 has been enabled as per the following document:

Enable TLSv1.0 on vCenter and ESXi Connections from View Compose

https://docs.vmware.com/en/VMware-Horizon-7/7.4/horizon-upgrades/GUID-1845E4AD-E84A-4304-A7DD-83170B8D21C5.html

Reviewing the event logs on the View Connection server show that the following error is logged:

Log Name: Application
Source: VMware

Event ID: 105

Level: Error

User: System

BROKER_SVI_CERT_INVALID

Certificate is invalid for Composer at address https://contosodrvc01.contoso.com:18443

Attributes:

            Node=contosoUKVV01.contoso.com

            ComposerId=https://contosodrvc01.contoso.com:18443

            Severity=ERROR

            Time=Fri Feb 02 22:01:58 GMT 2018

            Module=Broker

            Source=com.vmware.vdi.desktopcontroller.PublishVcCertToSviFederatedTask

            Acknowledged=true

clip_image002

The Events in the View Administrator console display the following message:

Certificate is invalid for Composer at address https://<vcenter>.FQDN.com:18443

clip_image002[5]

Attempting to view the properties of a linked-clone pool displays the following error:

The certificate configured on View Comopser Server is invalid, blocking communication with this server. To resume communication, replace the certificate with a valid certificate signed by a CA. Alternatively, accept the certificate thumbprint by clicking Verify in the View Administrator dashboard.

clip_image002[7]

You’ve confirmed that the correct self-signed certificate is assigned to the VMware Horizon View Composer service by executing the command:

sviconfig -operation=ReplaceCertificate -delete=false

clip_image002[9]

You’ve tried changing the pae-SVIURL to the short NetBIOS name instead of the FQDN:

clip_image002[11]

Solution

If you do not include on issuing a certificate from a trusted Certificate Authority and would like to use the self-signed certificate, the way to accept it is to navigate to the View Configuration > Servers menu, select the vCenter instance then click Edit to bring up the vCenter Server settings, then click Edit under the View Composer Server Settings section:

image

Getting into the configuration settings of the View Composer will now display the following invalid certificate detected prompt:

clip_image002[13]

Click on the View Certificate… button will display the option to accept the self-signed certificate:

clip_image002[15]

Accepting the certificate should now display the View Composer Servers health as green:

clip_image002[17]

Saturday, January 6, 2018

Patching a VMware ESXi 5.5 host offline via CLI

One of the more common questions I’m asked every year is how to patch an ESXi host without using VMware Update Manager because it is quite common for ESXi hosts as well as vCenter to not have internet access. What I typically do when asked is point them to one of my previous posts where I demonstrated the process for a Nutanix infrastructure:

Upgrading Nutanix blades from ESXi 5.1 to 5.5
http://terenceluk.blogspot.com/2014/12/upgrading-nutanix-blades-from-esxi-51.html

… because the post includes the command for non-Nutanix deployments at the end but since I’ve received follow up questions about the process, I thought it would be a good idea to write a post that specifically demonstrates this for any ESXi deployments.

Step #1 – Identify and download desired ESXi build level

Begin by identifying the which build you’d like to patch the ESXi to:

https://kb.vmware.com/s/article/2143832

image

With the build number identified, proceed to the following URL to locate and download the patch:

https://my.vmware.com/group/vmware/patch#search

image

image

Download the patch:

image

The package you download should be a ZIP file containing files similar to the following screenshot:

vib20

index.xml

metadata.zip

vendor-index.xml

image

Step #2 – Upload patch to datastore

Upload the package to a datastore that the host you’re patching has access to by using a utility such as WinSCP:

image

Step #3 – Patch ESXi host via CLI

Proceed to either access the console directly on the server or SSH to it.

image

The command we’ll be using to install the patch will look as such:

esxcli software vib install -d <full path to patch file>

Note that it is important to specify the full path to the patch file because if you don’t then the follow error will be thrown:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

-d ESXi550-201612001.zip

[MetadataDownloadError]

Could not download from depot at zip:/var/log/vmware/ESXi550-201612001.zip?index.xml, skipping (('zip:/var/log/vmware/ESXi550-201612001.zip?index.xml', '', "Error extracting index.xml from /var/log/vmware/ESXi550-201612001.zip: [Errno 2] No such file or directory: '/var/log/vmware/ESXi550-201612001.zip'"))

url = zip:/var/log/vmware/ESXi550-201612001.zip?index.xml

Please refer to the log file for more details.

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea #

image

Forgetting to include the -d will throw the following error:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

/vmfs/volumes/rvbd_vsp_datastore/ESXi550-201612001.zip

Error: Unknown command or namespace software vib install /vmfs/volumes/rvbd_vsp_datastore/ESXi550-201612001.zip

image

The patch will successfully install if the full path is specified and the output would look as such:

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea # esxcli software vib install

-d /vmfs/volumes/rvbd_vsp_datastore/ESXi550-201612001.zip

Installation Result

Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Reboot Required: true

VIBs Installed: VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.550.3.95.4345813, VMware_bootbank_esx-base_5.5.0-3.100.4722766, VMware_bootbank_esx-tboot_5.5.0-3.100.4722766, VMware_bootbank_esx-ui_1.12.0-4722375, VMware_bootbank_lsi-msgpt3_00.255.03.03-2vmw.550.3.78.3248547, VMware_bootbank_misc-drivers_5.5.0-3.95.4345813, VMware_bootbank_net-e1000e_3.2.2.1-2vmw.550.3.78.3248547, VMware_bootbank_net-igb_5.0.5.1.1-1vmw.550.2.54.2403361, VMware_bootbank_net-ixgbe_3.7.13.7.14iov-12vmw.550.2.62.2718055, VMware_bootbank_sata-ahci_3.0-22vmw.550.3.89.4179633, VMware_bootbank_xhci-xhci_1.0-3vmw.550.3.95.4345813, VMware_locker_tools-light_5.5.0-3.92.4345810

VIBs Removed: Intel_bootbank_net-igb_5.2.7-1OEM.550.0.0.1331820, Intel_bootbank_net-ixgbe_3.21.4-1OEM.550.0.0.1331820, VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.550.0.0.1331820, VMware_bootbank_esx-base_5.5.0-3.71.3116895, VMware_bootbank_esx-tboot_5.5.0-2.33.2068190, VMware_bootbank_lsi-msgpt3_00.255.03.03-1vmw.550.1.15.1623387, VMware_bootbank_misc-drivers_5.5.0-3.68.3029944, VMware_bootbank_net-e1000e_3.2.2.1-2vmw.550.3.68.3029944, VMware_bootbank_sata-ahci_3.0-22vmw.550.3.68.3029944, VMware_bootbank_xhci-xhci_1.0-2vmw.550.3.68.3029944, VMware_locker_tools-light_5.5.0-3.68.3029944

VIBs Skipped: VMware_bootbank_ata-pata-amd_0.3.10-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-atiixp_0.4.6-4vmw.550.0.0.1331820, VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-serverworks_0.4.3-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.550.0.0.1331820, VMware_bootbank_ata-pata-via_0.3.3-2vmw.550.0.0.1331820, VMware_bootbank_block-cciss_3.6.14-10vmw.550.0.0.1331820, VMware_bootbank_elxnet_10.2.309.6v-1vmw.550.3.68.3029944, VMware_bootbank_esx-dvfilter-generic-fastpath_5.5.0-0.0.1331820, VMware_bootbank_esx-xlibs_5.5.0-0.0.1331820, VMware_bootbank_esx-xserver_5.5.0-0.0.1331820, VMware_bootbank_ima-qla4xxx_2.01.31-1vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-devintf_39.1-4vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.550.0.0.1331820, VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.550.0.0.1331820, VMware_bootbank_lpfc_10.0.100.1-1vmw.550.0.0.1331820, VMware_bootbank_lsi-mr3_0.255.03.01-2vmw.550.3.68.3029944, VMware_bootbank_misc-cnic-register_1.72.1.v50.1i-1vmw.550.0.0.1331820, VMware_bootbank_mtip32xx-native_3.3.4-1vmw.550.1.15.1623387, VMware_bootbank_net-be2net_4.6.100.0v-1vmw.550.0.0.1331820, VMware_bootbank_net-bnx2_2.2.3d.v55.2-1vmw.550.0.0.1331820, VMware_bootbank_net-bnx2x_1.72.56.v55.2-1vmw.550.0.0.1331820, VMware_bootbank_net-cnic_1.72.52.v55.1-1vmw.550.0.0.1331820, VMware_bootbank_net-e1000_8.0.3.1-3vmw.550.0.0.1331820, VMware_bootbank_net-enic_1.4.2.15a-1vmw.550.0.0.1331820, VMware_bootbank_net-forcedeth_0.61-2vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.550.0.0.1331820, VMware_bootbank_net-nx-nic_5.0.621-1vmw.550.0.0.1331820, VMware_bootbank_net-tg3_3.123c.v55.5-1vmw.550.2.33.2068190, VMware_bootbank_net-vmxnet3_1.1.3.0-3vmw.550.2.39.2143827, VMware_bootbank_ohci-usb-ohci_1.0-3vmw.550.0.0.1331820, VMware_bootbank_qlnativefc_1.0.12.0-1vmw.550.0.0.1331820, VMware_bootbank_rste_2.0.2.0088-4vmw.550.1.15.1623387, VMware_bootbank_sata-ata-piix_2.12-10vmw.550.2.33.2068190, VMware_bootbank_sata-sata-nv_3.5-4vmw.550.0.0.1331820, VMware_bootbank_sata-sata-promise_2.12-3vmw.550.0.0.1331820, VMware_bootbank_sata-sata-sil24_1.1-1vmw.550.0.0.1331820, VMware_bootbank_sata-sata-sil_2.3-4vmw.550.0.0.1331820, VMware_bootbank_sata-sata-svw_2.3-3vmw.550.0.0.1331820, VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.550.0.0.1331820, VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.550.0.0.1331820, VMware_bootbank_scsi-aic79xx_3.1-5vmw.550.0.0.1331820, VMware_bootbank_scsi-bnx2fc_1.72.53.v55.1-1vmw.550.0.0.1331820, VMware_bootbank_scsi-bnx2i_2.72.11.v55.4-1vmw.550.0.0.1331820, VMware_bootbank_scsi-fnic_1.5.0.4-1vmw.550.0.0.1331820, VMware_bootbank_scsi-hpsa_5.5.0-44vmw.550.0.0.1331820, VMware_bootbank_scsi-ips_7.12.05-4vmw.550.0.0.1331820, VMware_bootbank_scsi-lpfc820_8.2.3.1-129vmw.550.0.0.1331820, VMware_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.550.0.0.1331820, VMware_bootbank_scsi-megaraid-sas_5.34-9vmw.550.3.68.3029944, VMware_bootbank_scsi-megaraid2_2.00.4-9vmw.550.0.0.1331820, VMware_bootbank_scsi-mpt2sas_14.00.00.00-3vmw.550.3.68.3029944, VMware_bootbank_scsi-mptsas_4.23.01.00-9vmw.550.3.68.3029944, VMware_bootbank_scsi-mptspi_4.23.01.00-9vmw.550.3.68.3029944, VMware_bootbank_scsi-qla2xxx_902.k1.1-12vmw.550.3.68.3029944, VMware_bootbank_scsi-qla4xxx_5.01.03.2-6vmw.550.0.0.1331820, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.550.0.0.1331820

/vmfs/volumes/58da7103-7810a474-5615-000eb6b4b6ea #

imageimage

Proceed to vMotion the VMs on this host to another host or shut them down if they don’t need to be up and restart the host:

image

The new build number should be displayed once the host has successfully restarted:

image

Wednesday, December 20, 2017

Unable to expand virtual machine hard disk that is backed up by VMware VDP

Problem

You have a virtual machine in your vSphere environment that you would like to expand one of the hard disks but as you browse to the hard disk that needs to get expanded, you see that it is locked:

image

You’ve checked to confirm that there are no snapshots on the virtual machine:

image

Reviewing the files for the virtual machine show that there are CTK files for the disks that are locked and unable to be expanded in capacity:

image

Reviewing the disk for the VM in the vSphere Web Client displays the same behavior:

image

Shutting down and reviewing the VM’s summary tab indicate that the disks need to be consolidated:

Configuration Issues

Virtual machine disks consolidation is needed.

image

Attempting to consolidate the disks throws the following error:

Consolidate virtual machine disk files

Unable to access file since it is locked

See the error stack for details on the cause of this problem.

Error Stack

An error occurred while consolidating disks: msg.snapshot.error-DISKLOCKED.

image

image

Solution

One of the reasons why this error would be thrown is if you have this virtual machine configured to be backed up by a VMware VDP appliance and the appliance hasn’t released the disk at the end of a backup job or perhaps it did not complete properly.  To determine whether this is the case, open the settings of the VDP appliance and review whether this virtual machine’s disk is attached to it as such:

image

You can verify the disks that belong to the VDR appliance by reviewing the administration console as such:

image

To correct the problem, simply remove the disk from the VDP appliance and select the Remove from virtual machine when prompted with the Removal Options (this does not delete the disk):

image

Note that the disk will still be locked at this point:

image

Proceed to consolidate the disks:

image

The disk should be unlocked once the consolidation task completes.

Wednesday, December 13, 2017

Attempting to launch Citrix XenApp / XenDesktop 7.x desktop or application published by NetScaler fails with the error: “The connection to “XenApp Desktop” failed with status (Unknown client error 1110).”

Problem

You attempt to launch a XenApp or XenDesktop published application or desktop published by NetScaler but receive the following error message:

The connection to “XenApp Desktop” failed with status (Unknown client error 1110).

image

Solution

One of the possible causes of this error message is if you have incorrect or have not configured any STA (Secure Ticket Authority) on your NetScalers.  Navigate to NetScaler Gateway > Virtual Servers, open the properties of the virtual server, scroll down to Published Applications and ensure STA Servers are configured:

image

Monday, December 4, 2017

Attempting to run Windows update on a Windows 7 desktop fails with the error code 80248015

Problem

You attempt to run Windows updates on a Windows 7 desktop but notice that it fails with the following message:

Windows could not search for new updates

An error occurred while checking for new updates for your computer.

Error(s) found:

Code 80248015

Windows Update encountered an unknown error.

clip_image002

Solution

One of the ways to correct this issue is to navigate to the Windows directory and rename the SoftwareDistribution folder so that it would get recreated. 

C:\Windows\SoftwareDistribution

clip_image002[4]

Restart the desktop once the folder has been renamed and Windows update should now work as expected:

clip_image002[6]