Posted on

Azure Stack Hotfix 1.1905.3.48

Applies to: Azure Stack Integrated Systems

Summary

This update for Azure Stack version 1905 includes fixes to storage and compute services. No new features are being introduced in this update.

Important changes include the following:

  • Fixed issue: Queue message expiration due to time skew in storage service.
  • Fixed issue: User image creation and deletion operations fail if an image is in a failed state. 

Symptoms

  • Queue messages: Messages in a queue will expire immediately after creation and tagged for garbage collection. This issue may present in three cases:
    • Retrieval of resources in Azure Resource Manager will fail.
    • Usage metric data is not collected and uploaded for processing.
    • Customer developed applications will fail to retrieve messages.
  • User image management: Failed user image creation will put the user image service into a bad state.
    • New image creation will fail once the service is in this state.
    • Deletion of image may fail with the following error “Error: An internal disk management error occurred.”

Cause

  • Queue message: Time skew in one of the instances running the table service will cause all subsequent messages to set to a fixed creation date. After 7 days (the default a expiration time for a message), the service will consider the message expired and tag it for garbage collection. 
  • User image management: Due to the failed image creation, the user image manager does not have sufficient data in its repository to delete the image.

Hotfix information

To apply this hotfix, you must have version 1.1905.0.40 installed.

Important As outlined in the Release Notes for the 1905 update, make sure that you refer to the update activity checklist on running Test-AzureStack (with specified parameters), and resolve any operational issues that are found, including all warnings and failures. Also, review active alerts and resolve any that require action.

File information

How to get this update

Download the following files. Then, follow the instructions on the Apply updates in Azure Stack page on the Microsoft Docs website to apply this update to Azure Stack.

Download the file now.
Download the hotfix .xml file now.

Posted on

Azure Stack 1905 update

Applies to: Azure Stack integrated systems
Source: Azure Stack 1905 Update

Summary

This article describes the contents of the 1905 update package. The update includes what’s new improvements, and fixes for this release of Azure Stack.

Important
This update package is only for Azure Stack integrated systems. Do not apply this update package to the Azure Stack Development Kit!

Build reference

The Azure Stack 1905 update build number is 1.1905.0.40.

Update type

The Azure Stack 1905 update build type is Full. As a result, the 1905 update has a longer runtime than express updates like 1903 and 1904. Exact runtimes for full updates typically depend on the number of nodes that your Azure Stack instance contains, the capacity used on your system by tenant workloads, your system’s network connectivity (if connected to the internet), and your system hardware configuration. The 1905 update has had the following expected runtimes in our internal testing: 4 nodes – 35 hours, 8 nodes – 45 hours, 12 nodes – 55 hours, 16 nodes – 70 hours. 1905 runtimes lasting longer than these expected values are not uncommon and do not require action by Azure Stack operators unless the update fails. For more information about update build types, see Manage updates in Azure Stack.

What’s in this update

  • With this update, the update engine in Azure Stack can update the firmware of scale unit nodes. This requires a compliant update package from the hardware partners. Reach out to your hardware partner for details about availability.
  • Windows Server 2019 is now supported and available to syndicate through the Azure Stack Marketplace. With this update, Windows Server 2019 can now be successfully activated on a 2016 host.
  • A new Azure Account Visual Studio Code extension allows developers to target Azure Stack by logging in and viewing subscriptions, as well as a number of other services. The Azure Account extension works on both Azure Active Directory (Azure AD) and AD FS environments, and only requires a small change in Visual Studio Code user settings. Visual Studio Code requires a service principal to be given permission in order to run on this environment. To do so, import the identity script and run the cmdlets specified in Multi-tenancy in Azure Stack. This requires an update to the home directory, and registration of the Guest tenant directory for each directory. An alert is displayed after updating to 1905 or later, to update the home directory tenant for which the Visual Studio Code service principal is included.

Improvements

  • As a part of enforcing TLS 1.2 on Azure Stack, the following extensions have been updated to these versions:
    • microsoft.customscriptextension-arm-1.9.3
    • microsoft.iaasdiagnostics-1.12.2.2
    • microsoft.antimalware-windows-arm-1.5.5.9
    • microsoft.dsc-arm-2.77.0.0
    • microsoft.vmaccessforlinux-1.5.2Please download these versions of the extensions immediately, so that new deployments of the extension do not fail when TLS 1.2 is enforced in a future release. Always set autoUpgradeMinorVersion=true so that minor version updates to extensions (for example, 1.8 to 1.9) are automatically performed.
  • A new Help and Support Overview in the Azure Stack portal makes it easier for operators to check their support options, get expert help, and learn more about Azure Stack. On integrated systems, creating a support request will preselect Azure Stack service. We highly recommend that customers use this experience to submit tickets rather than using the global Azure portal. For more information, see Azure Stack Help and Support.
  • When multiple Azure Active Directories are onboarded (through this process), it is possible to neglect rerunning the script when certain updates occur, or when changes to the Azure AD Service Principal authorization cause rights to be missing. This can cause various issues, from blocked access for certain features, to more discrete failures which are hard to trace back to the original issue. To prevent this, 1905 introduces a new feature that checks for these permissions and creates an alert when certain configuration issues are found. This validation runs every hour, and displays the remediation actions required to fix the issue. The alert closes once all the tenants are in a healthy state.
  • Improved reliability of infrastructure backup operations during service failover.
  • A new version of the Azure Stack Nagios plugin is available that uses the Azure Active Directory authentication libraries (ADAL) for authentication. The plugin now also supports Azure AD and Active Directory Federation Services (AD FS) deployments of Azure Stack. For more information, see the Nagios plugin exchange site.
  • A new hybrid profile 2019-03-01-Hybrid was released that supports all the latest features in Azure Stack. Both Azure PowerShell and Azure CLI support the 2019-03-01-Hybrid profile. The .NET, Ruby, Node.js, Go, and Python SDKs have published packages that support the 2019-03-01-Hybrid profile. The respective documentation and some samples have been updated to reflect the changes.
  • The Node.js SDK now supports API profiles. Packages that support the 2019-03-01-Hybrid profile are published.
  • The 1905 Azure Stack update adds two new infrastructure roles to improve platform reliability and supportability:
    • Infrastructure ring: In the future, the infrastructure ring will host containerized versions of existing infrastructure roles – for example, xrp – that currently require their own designated infrastructure VMs. This will improve platform reliability and reduce the number of infrastructure VMs that Azure Stack requires. This subsequently reduces the overall resource consumption of Azure Stack’s infrastructure roles in the future.
    • Support ring: In the future, the support ring will be used to handle enhanced support scenarios for customers.In addition, we added an extra instance of the domain controller VM for improved availability of this role.These changes will increase the resource consumption of Azure Stack infrastructure in the following ways:Azure Stack SKUIncrease in Compute ConsumptionIncrease in Memory Consumption4 Nodes22 vCPU28 GB8 Nodes38 vCPU44 GB12 Nodes54 vCPU60 GB16 Nodes70 vCPU76 GB

Changes

  • To increase reliability and availability during planned and unplanned maintenance scenarios, Azure Stack adds an additional infrastructure role instance for domain services.
  • With this update, during repair and add node operations, the hardware is validated to ensure homogenous scale unit nodes within a scale unit.
  • If scheduled backups are failing to complete and the defined retention period is exceeded, the infrastructure backup controller will ensure at least one successful backup is retained.

Fixes

  • Fixed an issue in which a Compute host agent warning appeared after restarting a node in the scale unit.
  • Fixed issues in marketplace management in the administrator portal which showed incorrect results when filters were applied, and showed duplicate publisher names in the publisher filter. Also made performance improvements to display results faster.
  • Fixed issue in the available backup blade that listed a new available backup before it completed upload to the external storage location. Now the available backup will show in the list after it is successfully uploaded to the storage location.
  • Fixed issue with retrieving recovery keys during backup operation.
  • Fixed issue with OEM update displaying version as ‘undefined’ in operator portal.

Security updates

For information about security updates in this update of Azure Stack, see Azure Stack security updates.

Update planning

Before applying the update, make sure to review the following information:

Download the update

You can download the Azure Stack 1905 update package from the Azure Stack download page. When using the downloader tool, be sure to use the latest version and not a cached copy from your downloads directory.

Hotfixes

Azure Stack releases hotfixes on a regular basis. Be sure to install the latest Azure Stack hotfix for 1904 before updating Azure Stack to 1905.

Azure Stack hotfixes are only applicable to Azure Stack integrated systems; do not attempt to install hotfixes on the ASDK.

Before applying the 1905 update

The 1905 release of Azure Stack must be applied on the 1904 release with the following hotfixes:

After successfully applying the 1905 update

After the installation of this update, install any applicable hotfixes. For more information, see Microsofts’ servicing policy.

Automatic update notifications

Customers with systems that can access the internet from the infrastructure network will see the Update available message in the operator portal. Systems without internet access can download and import the .zip file with the corresponding .xml.

Tip
Subscribe to the following RSS or Atom feeds to keep up with Azure Stack hotfixes:

Known issues (post installation)

This article lists known issues in the 1905 release of Azure Stack. The list is updated as new issues are identified.

Important
Review this section before applying the update!

Update process

Host node update prerequisite failure

  • Applicable: This issue applies to the 1905 update.
  • Cause: When attempting to install the 1905 Azure Stack update, the status for the update might fail due to Host Node Update Prerequisite. This is generally caused by a host node having insufficient free disk space.
  • Remediation: Contact Azure Stack support to receive assistance clearing disk space on the host node.
  • Occurrence: Uncommon

Preparation failed

  • Applicable: This issue applies to all supported releases.
  • Cause: When attempting to install the 1905 Azure Stack update, the status for the update might fail and change state to PreparationFailed. This is caused by the update resource provider (URP) being unable to properly transfer the files from the storage container to an internal infrastructure share for processing. The 1905 update package is larger than previous update packages which may make this issue more likely to occur.
  • Remediation: Starting with version 1901 (1.1901.0.95), you can work around this issue by clicking Update now again (not Resume). The URP then cleans up the files from the previous attempt, and restarts the download. If the problem persists, we recommend manually uploading the update package by following the Import and install updates section.
  • Occurrence: Common

Portal

Subscription resources

  • Applicable: This issue applies to all supported releases.
  • Cause: Deleting user subscriptions results in orphaned resources.
  • Remediation: First delete user resources or the entire resource group, and then delete the user subscriptions.
  • Occurrence: Common

Subscription permissions

  • Applicable: This issue applies to all supported releases.
  • Cause: You cannot view permissions to your subscription using the Azure Stack portals.
  • Remediation: Use PowerShell to verify permissions.
  • Occurrence: Common

Marketplace management

  • Applicable: This issue applies to 1904 and 1905
  • Cause: The marketplace management screen is not visible when you sign in to the administrator portal.
  • Remediation: Refresh the browser or go to Settings and select the option Reset to default settings.
  • Occurrence: Intermittent

Docker extension

  • Applicable: This issue applies to all supported releases.
  • Cause: In both the administrator and user portals, if you search for Docker, the item is incorrectly returned. It is not available in Azure Stack. If you try to create it, an error is displayed.
  • Remediation: No mitigation.
  • Occurrence: Common

Upload blob

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, when you try to upload a blob using the OAuth(preview) option, the task fails with an error message.
  • Remediation: Upload the blob using the SAS option.
  • Occurrence: Common

Template

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, the template deployment UI does not populate parameters for the template names beginning with “_” (the underscore character).
  • Remediation: Remove the “_” (underscore character) from the template name.
  • Occurrence: Common

Networking

Load balancer

Add backend pool

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, if you attempt to add a Backend Pool to a Load Balancer, the operation fails with the error message failed to update Load Balancer….
  • Remediation: Use PowerShell, CLI or a Resource Manager template to associate the backend pool with a load balancer resource.
  • Occurrence: Common

Create inbound NAT

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, if you attempt to create an Inbound NAT Rule for a Load Balancer, the operation fails with the error message Failed to update Load Balancer….
  • Remediation: Use PowerShell, CLI or a Resource Manager template to associate the backend pool with a load balancer resource.
  • Occurrence: Common

Create load balancer

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, the Create Load Balancer window shows an option to create a Standard load balancer SKU. This option is not supported in Azure Stack.
  • Remediation: Use the Basic load balancer options instead.
  • Occurrence: Common

Public IP address

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, the Create Public IP Address window shows an option to create a Standard SKU. The Standard SKU is not supported in Azure Stack.
  • Remediation: Use the Basic SKU for public IP address.
  • Occurrence: Common

Compute

VM boot diagnostics

  • Applicable: This issue applies to all supported releases.
  • Cause: When creating a new Windows virtual machine (VM), the following error may be displayed: Failed to start virtual machine ‘vm-name’. Error: Failed to update serial output settings for VM ‘vm-name’. The error occurs if you enable boot diagnostics on a VM, but delete your boot diagnostics storage account.
  • Remediation: Recreate the storage account with the same name you used previously.
  • Occurrence: Common

VM resize

  • Applicable: This issue applies to the 1905 release.
  • Cause: Unable to successfully resize a managed disk VM. Attempting to resize the VM generates an error with “code”: “InternalOperationError”, “message”: “An internal error occurred in the operation.”
  • Remediation: We are working to remediate this in the next release. Currently, you must recreate the VM with the new VM size.
  • Occurrence: Common

Virtual machine scale set

CentOS

  • Applicable: This issue applies to all supported releases.
  • Cause: The virtual machine scale set creation experience provides CentOS-based 7.2 as an option for deployment. CentOS 7.2 is not available on Azure Stack Marketplace which will cause deployment failures calling out that the image is not found.
  • Remediation: Select another operating system for your deployment, or use an Azure Resource Manager template specifying another CentOS image that has been downloaded prior to deployment from the marketplace by the operator.
  • Occurrence: Common

Remove scale set

  • Applicable: This issue applies to all supported releases.
  • Cause: You cannot remove a scale set from the Virtual machine scale sets blade.
  • Remediation: Select the scale set that you want to remove, then click the Delete button from the Overview pane.
  • Occurrence: Common

Create failures during patch and update on 4-node Azure Stack environments

  • Applicable: This issue applies to all supported releases.
  • Cause: Creating VMs in an availability set of 3 fault domains and creating a virtual machine scale set instance fails with a FabricVmPlacementErrorUnsupportedFaultDomainSize error during the update process on a 4-node Azure Stack environment.
  • Remediation: You can create single VMs in an availability set with 2 fault domains successfully. However, scale set instance creation is still not available during the update process on a 4-node Azure Stack.

Scale set instance view blade doesn’t load

  • Applicable: This issue applies to 1904 and 1905 release.
  • Cause: The instance view blade of a virtual machine scale set located at Azure Stack portal -> Dashboard -> Virtual machine scale sets -> AnyScaleSet – Instances -> AnyScaleSetInstance fails to load, and displays a crying cloud image.
  • Remediation: There is currently no remediation and we are working on a fix. Until then, please use the CLI command az vmss get-instance-view to get the instance view of a scale set.

Ubuntu SSH access

  • Applicable: This issue applies to all supported releases.
  • Cause: An Ubuntu 18.04 VM created with SSH authorization enabled does not allow you to use the SSH keys to sign in.
  • Remediation: Use VM access for the Linux extension to implement SSH keys after provisioning, or use password-based authentication.
  • Occurrence: Common

Next steps

Posted on

Azure Stack Hotfix 1.1904.4.45

Applies to: Azure Stack integrated systems

Summary

This update for Azure Stack version 1904 includes fixes and reliability improvements. No new features are being introduced in this update.

Important changes include the following:

  • Fixed issue: Compute host agent warning appears after restarting a node in the scale unit. 
  • Fixed issue: Infrastructure backup fails to back up internal identity service. 
  • Improved reliability of Update Resource provider automatic downloader.

Symptoms

  • Compute host agent: Warning appears after restarting a node in the scale unit. 
  • Identity service backup: Infrastructure backup finishes but is only partially successful because it does not back up the internal identity service.

Cause

  • Compute host agent: Restarting the node changes the default startup setting for the compute host agent service.
  • Identity service backup: There is a bug in the backup workflow.

Hotfix information

To apply this hotfix, you must have version 1.1904.0.36 installed.

Important! As outlined in the Release Notes for the 1904 update, make sure that you refer to the update activity checklist on running Test-AzureStack (with specified parameters), and resolve any operational issues that are found, including all warnings and failures. Also, review active alerts and resolve any that require action.

File information

How to get this update

Download the following files. Then, follow the instructions on the Apply updates in Azure Stack page on the Microsoft Docs website to apply this update to Azure Stack.

Download the file now.
Download the hotfix .xml file now.

Posted on

Azure Stack 1904 Update

Applies to: Azure Stack integrated systems

This article describes the contents of the 1904 update package. The update includes what’s new improvements, and fixes for this release of Azure Stack. This article contains the following information:

Important!

This update package is only for Azure Stack integrated systems. Do not apply this update package to the Azure Stack Development Kit.

Build reference

The Azure Stack 1904 update build number is 1.1904.0.36.

What’s in this update

Improvements

  • Significant improvements have been made to the Software Defined Networking (SDN) Stack in 1904. These improvements increase the overall servicing and reliability of the SDN stack in Azure Stack.
  • Added a notification in the administrator portal, when the currently logged in user does not have the necessary permissions, which enables the dashboard to load properly. It also contains a link to the documentation that explains which accounts have the appropriate permissions, depending on the identity provider used during deployment.
  • Added improvements to VM resiliency and uptime, which resolves the scenario in which all VMs go offline if the storage volume containing the VM configuration files goes offline.
  • Added optimization to the number of VMs evacuated concurrently and placed a cap on bandwidth consumed, to address VM brownouts or blackouts if the network is under heavy load. This change increases VM uptime when a system is updating.
  • Improved resource throttling when a system is running at scale to protect against internal processes exhausting platform resources, resulting in failed operations in the portal.
  • Improved filtering capabilities enable operators to apply multiple filters at the same time. You can only sort on the Namecolumn in the new user interface.
  • Improvements to the process of deleting offers, plans, quotas, and subscriptions. You can now successfully delete offers, quotas, plans, and subscriptions from the Administrator portal if the object you want to delete has no dependencies. For more information, see this article.
  • Improved syslog message volume by filtering out unnecessary events and providing a configuration parameter to select desired severity level for forwarded messages. For more information about how to configure the severity level, see Azure Stack datacenter integration – syslog forwarding.
  • The Azure Stack Infrastructure consumes an additional 12 GB + (4 GB * Number of Azure Stack hosts) from the 1904 update onwards. This means that in a 4 node stamp there will be an additional capacity consumption of 28 GB (12 GB + 4 GB * 4) reflected in the capacity screen of the Azure Stack administrator portal. Your update to the 1904 release should succeed even if the additional memory consumption puts your Azure Stack stamp over capacity. If your Azure Stack stamp is over memory usage AFTER the update is completed, you will see an alert reflecting this state, with remediation steps to de-allocate some VMs.
  • Added a new capability to the Get-AzureStackLog cmdlet by incorporating an additional parameter, -OutputSASUri. You can now collect Azure Stack logs from your environment and store them in the specified Azure Storage blob container. For more information, see Azure Stack diagnostics.
  • Added a new memory check in the Test-AzureStack UpdateReadiness group, which checks to see if you have enough memory available on the stack for the update to complete successfully.
  • Improvements to Test-AzureStack for evaluating Service Fabric health.
  • Improvements to hardware updates, which reduces the time it takes to complete drive firmware update to 2-4 hours. The update engine dynamically determines which portions of the update need to execute, based on content in the package.
  • Added robust operation prechecks to prevent disruptive infrastructure role instance operations that affect availability.
  • Improvements to idempotency of infrastructure backup action plan.
  • Improvements to Azure Stack log collection. These improvements reduce the time it takes to retrieve the set of logs. Also, the Get-AzureStackLog cmdlet no longer generates default logs for the OEM role. You must execute the Invoke-AzureStackOnDemandLog cmdlet, specifying the role to retrieve the OEM logs. For more information , see Azure Stack diagnostics.
  • Azure Stack now monitors the federation data URL provided for datacenter integration with ADFS. This improves reliability during secret rotation of the customer ADFS instance or farm.

Changes

  • Removed the option for Azure Stack operators to shut down infrastructure role instances in the administrator portal. The restart functionality ensures a clean shutdown attempt before restarting the infrastructure role instance. For advanced scenarios, the API and PowerShell functionality remains available.
  • There is a new Marketplace management experience, with separate screens for Marketplace images and resource providers. For now, the Resource providers window is empty, but in future releases new PaaS service offerings will appear and be managed in the Resource providers window.
  • Changes to the update experience in the operator portal. There is a new grid for resource provider updates. The ability to update resource providers is not available yet.
  • Changes to the update installation experience in the operator portal. To help Azure Stack operators respond appropriately to an update issue, the portal now provides more specific recommendations based on the health of the scale unit, as derived automatically by running Test-AzureStack and parsing the results. Based on the result, it will inform the operator to take one of two actions:
    • A “soft” warning alert is displayed in the portal that reads “The most recent update needs attention. Microsoft recommends opening a service request during normal business hours. As part of the update process, Test-AzureStack is performed, and based on the output we generate the most appropriate alert. In this case, Test-AzureStack passed.”
    • A “hard” critical alert is displayed in the portal that reads, “The most recent update failed. Microsoft recommends opening a service request as soon as possible. As part of the update process, Test-AzureStack is performed, and based on the output we generate the most appropriate alert. In this case, Test-AzureStack also failed.”
  • Updated Azure Linux Agent version 2.2.38.0. This support allows customers to maintain consistent Linux images between Azure and Azure Stack.

Fixes

  • Fixed an issue in which the syslog configuration was not persisted through an update cycle, causing the syslog client to lose its configuration, and the syslog messages to stop being forwarded. Syslog configuration is now preserved.
  • Fixed an issue in CRP that blocked deallocation of VMs. Previously, if a VM contained multiple large managed disks, deallocating the VM might have failed with a timeout error.
  • Fixed issue with Windows Defender engine impacting access to scale-unit storage.
  • Fixed a user portal issue in which the Access Policy window for blob storage accounts failed to load.
  • Fixed an issue in both administrator and user portals, in which erroneous notifications about the global Azure portal were displayed.
  • Fixed a user portal issue in which selecting the Feedback tile caused an empty browser tab to open.
  • Fixed a portal issue in which changing a static IP address for an IP configuration that was bound to a network adapter attached to a VM instance, caused an error message to be displayed.
  • Fixed a user portal issue in which attempting to Attach Network Interface to an existing VM via the Networking window caused the operation to fail with an error message.
  • Fixed an issue in which Azure Stack did not support attaching more than 4 Network Interfaces (NICs) to a VM instance.
  • Fixed a portal issue in which adding an inbound security rule and selecting Service Tag as the source, displayed several options that are not available for Azure Stack.
  • Fixed the issue in which Network Security Groups (NSGs) did not work in Azure Stack in the same way as global Azure.
  • Fixed an issue in Marketplace management, which hides all downloaded products if registration expires or is removed.
  • Fixed an issue in which issuing a Set-AzureRmVirtualNetworkGatewayConnection command in PowerShell to an existing virtual network gateway connection failed with the error message Invalid shared key configured….
  • Fixed an issue that caused the Network Resource Provider (NRP) to be out of sync with the network controller, resulting in duplicate resources being requested. In some cases, this resulted in leaving the parent resource in an error state.
  • Fixed an issue in which if a user that was assigned a contributor role to a subscription, but was not explicitly given read permissions, an error was generated that read …The client ‘somelogonaccount@domain.com’ with object ID {GUID} does not have authorization to perform action… when attempting to save a change to a resource.
  • Fixed an issue in which the marketplace management screen was empty if the offline syndication tool was used to upload images, and any one of them was missing the icon URI(s).

Security updates

This update of Azure Stack does not include security updates to the underlying operating system which hosts Azure Stack. For information, see Azure Stack security updates.

Update planning

Before applying the update, make sure to review the following information:

Note!
Make sure to use the latest version of the Azure Stack Capacity Planner tool to perform your workload planning and sizing. The latest version contains bug fixes and provides new features that are released with each Azure Stack update.

Download the update

You can download the Azure Stack 1904 update package from the Azure Stack download page.

Hotfixes

Azure Stack releases hotfixes on a regular basis. Be sure to install the latest Azure Stack hotfix for 1903 before updating Azure Stack to 1904.

Azure Stack hotfixes are only applicable to Azure Stack integrated systems; do not attempt to install hotfixes on the ASDK.

Before applying the 1904 update

The 1904 release of Azure Stack must be applied on the 1903 release with the following hotfixes!

After successfully applying the 1904 update

After the installation of this update, install any applicable hotfixes. For more information, see our Servicing Policy.

  • No hotfixes available for 1904.

Automatic update notifications

Customers with systems that can access the internet from the infrastructure network will see the Update available message in the operator portal. Systems without internet access can download and import the .zip file with the corresponding .xml.

 Tip

Subscribe to the following RSS or Atom feeds to keep up with Azure Stack hotfixes:

Next steps

Posted on

Azure Stack Hotfix 1.1903.2.39

Applies to: Azure Stack Integrated Systems

Summary

This update for Azure Stack version 1903 includes reliability improvements. No new features are being introduced in this update. Important changes include the following:

  • Fix time out issue during stop-deallocate of IaaS VMs with multiple large managed disks.
  • Increase VM uptime when a system is updating.
  • Fix issues with resource exhaustion when the system is running at scale improving overall system stability. 
  • Update to Windows Defender engine to address issues with storage availability.  

Duration
The installation of this Hotfix took us about 1 hour and 10 minutes on a twelve node Azure Stack.

Symptoms

  • IaaS VM with multiple large managed disks: During stop-deallocate, the operation fails due to time out. 
  • VM uptime during system updates: Azure Stack updates perform a rolling update of the scale-unit by draining one node at a time and updating it. During node drain, the goal is to evacuate a node with no perceived downtime to the VM. However, if the network is under heavy load, VMs may incur a brownout or a blackout.  
  • Resource exhaustion: When a system is running at scale, internal processes may exhaust platform resources resulting in failed operations in the portal. 
  • Windows Defender: Issues in the Windows Defender engine impact the availability of storage.  

Cause

  • IaaS VM with multiple large managed disks: Missing information in CRP impacts the stop-deallocate operation. 
  • VM uptime during system updates: Number of concurrent VMs evacuate and bandwidth consumed is too high. 
  • Resource exhaustion: Sub-optimal Throttling of resource allocation for process execution. 
  • Windows Defender: Stability issues identified in the Windows Defender engine in this release.

Resolution

  • IaaS VM with multiple large managed disks: Fix in CRP to address the time out. 
  • VM uptime during system updates: Reduce the number of VMs evacuated concurrently and place a cap on bandwidth consumed. 
  • Resource exhaustion: Optimize resource throttling. 
  • Windows Defender: Update Windows Defender engine version. 

Hotfix information

To apply this hotfix, you must have version 1.1903.0.35 installed.

Important!
As outlined in the Release Notes for the 1903 update, be sure to follow the instructions in the Prerequisites section on running Test-AzureStack (with specified parameters) and resolve any operational issues found, including all warnings and failures. Also review active alerts and resolve any that require action.

File information

How to get this update

Download the following files. As soon as the files have been downloaded, follow the instructions on the Apply updates in Azure Stack page on the Microsoft Docs website to apply this update to Azure Stack.Download the hotfix .zip file now.

Download the hotfix .xml file now.

More information

Azure Stack update resources

Manage updates in Azure Stack overview

Apply updates in Azure Stack

Monitor updates in Azure Stack by using the privileged endpoint

Posted on

Azure Stack 1903 update

Applies to: Azure Stack integrated systems

This article describes the contents of the 1903 update package. The update includes improvements, fixes, and new features for this version of Azure Stack. This article also describes known issues in this release, and includes a link to download the update. Known issues are divided into issues directly related to the update process, and issues with the build (post-installation).

This update package is only for Azure Stack integrated systems. Do not apply this update package to the Azure Stack Development Kit.

Build reference

The Azure Stack 1903 update build number is 1.1903.0.35.

The 1903 payload does not include an ASDK release.

Hotfixes

Azure Stack releases hotfixes on a regular basis. Be sure to install the latest Azure Stack hotfix for 1902 before updating Azure Stack to 1903!

Azure Stack hotfixes are only applicable to Azure Stack integrated systems; do not attempt to install hotfixes on the ASDK!

Azure Stack hotfixes

Improvements

  • The 1903 update payload contains an update to components of Azure Stack that do not include the underlying operating system which hosts Azure Stack. This enables certain updates to be scoped. As a result, the expected time it takes for the 1903 update to complete is less (approx. 16 hours, but exact times can vary). This decrease in runtime is specific to the 1903 update and subsequent updates may contain updates to the operating system, implying different runtimes. Future updates will provide similar guidance on the expected time the update takes to complete, depending on the payload included.
  • Fixed a bug in networking that prevented changes to the idle timeout (minutes) value of a Public IP Address from taking effect. Previously, changes to this value were ignored, so that regardless of any changes you made, the value would default to 4 minutes. This setting controls how many minutes to keep a TCP connection open without relying on clients to send keep-alive messages. Note this bug only affected instance level public IPs, not public IPs assigned to a load balancer.
  • Improvements to the reliability of the update engine, including auto-remediation of common issues so that updates apply without interruption.
  • Improvements to the detection and remediation of low disk space conditions.

Secret management

  • Azure Stack now supports rotation of the root certificate used by certificates for external secret rotation. For more information, see this article.
  • 1903 contains performance improvements for secret rotation that reduce the time that it takes to execute internal secret rotation.

Prerequisites

Install the latest Azure Stack hotfix for 1902 (if any) before updating to 1903.

  • Make sure to use the latest version of the Azure Stack capacity planner to do your workload planning and sizing. The latest version contains bug fixes and provides new features that are released with each Azure Stack update.
  • Before you start installation of this update, run Test-AzureStack with the following parameter to validate the status of your Azure Stack and resolve any operational issues found, including all warnings and failures. Also review active alerts, and resolve any that require action: PowerShell
  • Test-AzureStack -Group UpdateReadiness
  • When Azure Stack is managed by System Center Operations Manager, make sure to update the Management Pack for Microsoft Azure Stack to version 1.0.3.11 before applying 1903.
  • The package format for the Azure Stack update has changed from .bin/.exe/.xml to .zip/.xml starting with the 1902 release. Customers with connected Azure Stack scale units will see the Update available message in the portal. Customers that are not connected can now simply download and import the .zip file with the corresponding .xml.

Known issues with the update process

  • When you run Test-AzureStack, a warning message from the Baseboard Management Controller (BMC) is displayed. You can safely ignore this warning.
  • During installation of this update, you might see alerts with the title **Error – Template for FaultType UserAccounts.New is missing.** You can safely ignore these alerts. The alerts close automatically after the installation of this update completes.

Post-update steps

Known issues (post-installation)

The following are post-installation known issues for this build version.

Portal

  • In both the administrator and user portals, if you search for “Docker,” the item is incorrectly returned. It is not available in Azure Stack. If you try to create it, a blade with an error indication is displayed.
  • Plans that are added to a user subscription as an add-on plan cannot be deleted, even when you remove the plan from the user subscription. The plan will remain until the subscriptions that reference the add-on plan are also deleted.
  • The two administrative subscription types that were introduced with version 1804 should not be used. The subscription types are Metering subscription, and Consumption subscription. These subscription types are visible in new Azure Stack environments beginning with version 1804 but are not yet ready for use. You should continue to use the Default Provider subscription type.
  • Deleting user subscriptions results in orphaned resources. As a workaround, first delete user resources or the entire resource group, and then delete the user subscriptions.
  • You cannot view permissions to your subscription using the Azure Stack portals. As a workaround, use PowerShell to verify permissions.
  • In the user portal, when you navigate to a blob within a storage account and try to open Access Policy from the navigation tree, the subsequent window fails to load.
  • In the user portal, when you try to upload a blob using the OAuth(preview) option, the task fails with an error message. To work around this issue, upload the blob using the SAS option.
  • When logged into the Azure Stack portals you might see notifications about the public Azure portal. You can safely ignore these notifications, as they do not currently apply to Azure Stack (for example, “1 new update – The following updates are now available: Azure portal April 2019 update”).

Compute

  • When creating a new Windows Virtual Machine (VM), the following error may be displayed: 'Failed to start virtual machine 'vm-name'. Error: Failed to update serial output settings for VM 'vm-name' The error occurs if you enable boot diagnostics on a VM but delete your boot diagnostics storage account. To work around this issue, recreate the storage account with the same name as you used previously.
  • The Virtual Machine Scale Set creation experience provides CentOS-based 7.2 as an option for deployment. Because that image is not available on Azure Stack, either select another operating system for your deployment, or use an Azure Resource Manager template specifying another CentOS image that has been downloaded prior to deployment from the marketplace by the operator.
  • After applying the 1903 update, you might encounter the following issues when deploying VMs with Managed Disks:
    • If the subscription was created before the 1808 update, deploying a VM with Managed Disks might fail with an internal error message. To resolve the error, follow these steps for each subscription:
      1. In the Tenant portal, go to Subscriptions and find the subscription. Select Resource Providers, then select Microsoft.Compute, and then click Re-register.
      2. Under the same subscription, go to Access Control (IAM), and verify that Azure Stack – Managed Disk is listed.
    • If you have configured a multi-tenant environment, deploying VMs in a subscription associated with a guest directory might fail with an internal error message. To resolve the error, follow these steps in this article to reconfigure each of your guest directories.
  • An Ubuntu 18.04 VM created with SSH authorization enabled will not allow you to use the SSH keys to sign in. As a workaround, use VM access for the Linux extension to implement SSH keys after provisioning, or use password-based authentication.
  • If you do not have a Hardware Lifecycle Host (HLH): before build 1902, you had to set group policy Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options to Send LM & NTLM – use NTLMv2 session security if negotiated. Since build 1902, you must leave it as Not Defined or set it to Send NTLMv2 response only (which is the default value). Otherwise, you won’t be able to establish a PowerShell remote session and you will see an Access is denied error: shell
  • PS C:\Users\Administrator> $session = New-PSSession -ComputerName x.x.x.x -ConfigurationName PrivilegedEndpoint -Credential $cred New-PSSession : [x.x.x.x] Connecting to remote server x.x.x.x failed with the following error message : Access is denied. For more information, see the about_Remote_Troubleshooting Help topic. At line:1 char:12 + $session = New-PSSession -ComputerName x.x.x.x -ConfigurationNa ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotingTransportException + FullyQualifiedErrorId : AccessDenied,PSSessionOpenFailed

Networking

  • In the Azure Stack portal, when you change a static IP address for an IP configuration that is bound to a network adapter attached to a VM instance, you will see a warning message that states The virtual machine associated with this network interface will be restarted to utilize the new private IP address... You can safely ignore this message; the IP address will be changed even if the VM instance does not restart.
  • In the portal, if you add an inbound security rule and select Service Tag as the source, several options are displayed in the Source Tag list that are not available for Azure Stack. The only options that are valid in Azure Stack are as follows:
    • Internet
    • VirtualNetwork
    • AzureLoadBalancer
    The other options are not supported as source tags in Azure Stack. Similarly, if you add an outbound security rule and select Service Tag as the destination, the same list of options for Source Tag is displayed. The only valid options are the same as for Source Tag, as described in the previous list.
  • Network security groups (NSGs) do not work in Azure Stack in the same way as global Azure. In Azure, you can set multiple ports on one NSG rule (using the portal, PowerShell, and Resource Manager templates). In Azure Stack however, you cannot set multiple ports on one NSG rule via the portal. To work around this issue, use a Resource Manager template or PowerShell to set these additional rules.
  • Azure Stack does not support attaching more than 4 Network Interfaces (NICs) to a VM instance today, regardless of the instance size.

App Service

  • You must register the storage resource provider before you create your first Azure Function in the subscription.

Download the update

You can download the Azure Stack 1903 update package from here.

In connected scenarios only, Azure Stack deployments periodically check a secured endpoint and automatically notify you if an update is available for your cloud. For more information, see managing updates for Azure Stack.

Next steps

Posted on

Welcome!

Hi and welcome to my site! Here I’ll try to share my day to day experiences with cloud computing and Azure and Azure Stack specifically. I hope to share my knowledge on these topics and maybe some other things as well.

If you like to know more about me, please refer to the About page.

I hope my blogs will be a useful source of information. If you have any questions or remarks, feel free to contact me.