Posted on

Azure Stack 1905 update

Applies to: Azure Stack integrated systems
Source: Azure Stack 1905 Update

Summary

This article describes the contents of the 1905 update package. The update includes what’s new improvements, and fixes for this release of Azure Stack.

Important
This update package is only for Azure Stack integrated systems. Do not apply this update package to the Azure Stack Development Kit!

Build reference

The Azure Stack 1905 update build number is 1.1905.0.40.

Update type

The Azure Stack 1905 update build type is Full. As a result, the 1905 update has a longer runtime than express updates like 1903 and 1904. Exact runtimes for full updates typically depend on the number of nodes that your Azure Stack instance contains, the capacity used on your system by tenant workloads, your system’s network connectivity (if connected to the internet), and your system hardware configuration. The 1905 update has had the following expected runtimes in our internal testing: 4 nodes – 35 hours, 8 nodes – 45 hours, 12 nodes – 55 hours, 16 nodes – 70 hours. 1905 runtimes lasting longer than these expected values are not uncommon and do not require action by Azure Stack operators unless the update fails. For more information about update build types, see Manage updates in Azure Stack.

What’s in this update

  • With this update, the update engine in Azure Stack can update the firmware of scale unit nodes. This requires a compliant update package from the hardware partners. Reach out to your hardware partner for details about availability.
  • Windows Server 2019 is now supported and available to syndicate through the Azure Stack Marketplace. With this update, Windows Server 2019 can now be successfully activated on a 2016 host.
  • A new Azure Account Visual Studio Code extension allows developers to target Azure Stack by logging in and viewing subscriptions, as well as a number of other services. The Azure Account extension works on both Azure Active Directory (Azure AD) and AD FS environments, and only requires a small change in Visual Studio Code user settings. Visual Studio Code requires a service principal to be given permission in order to run on this environment. To do so, import the identity script and run the cmdlets specified in Multi-tenancy in Azure Stack. This requires an update to the home directory, and registration of the Guest tenant directory for each directory. An alert is displayed after updating to 1905 or later, to update the home directory tenant for which the Visual Studio Code service principal is included.

Improvements

  • As a part of enforcing TLS 1.2 on Azure Stack, the following extensions have been updated to these versions:
    • microsoft.customscriptextension-arm-1.9.3
    • microsoft.iaasdiagnostics-1.12.2.2
    • microsoft.antimalware-windows-arm-1.5.5.9
    • microsoft.dsc-arm-2.77.0.0
    • microsoft.vmaccessforlinux-1.5.2Please download these versions of the extensions immediately, so that new deployments of the extension do not fail when TLS 1.2 is enforced in a future release. Always set autoUpgradeMinorVersion=true so that minor version updates to extensions (for example, 1.8 to 1.9) are automatically performed.
  • A new Help and Support Overview in the Azure Stack portal makes it easier for operators to check their support options, get expert help, and learn more about Azure Stack. On integrated systems, creating a support request will preselect Azure Stack service. We highly recommend that customers use this experience to submit tickets rather than using the global Azure portal. For more information, see Azure Stack Help and Support.
  • When multiple Azure Active Directories are onboarded (through this process), it is possible to neglect rerunning the script when certain updates occur, or when changes to the Azure AD Service Principal authorization cause rights to be missing. This can cause various issues, from blocked access for certain features, to more discrete failures which are hard to trace back to the original issue. To prevent this, 1905 introduces a new feature that checks for these permissions and creates an alert when certain configuration issues are found. This validation runs every hour, and displays the remediation actions required to fix the issue. The alert closes once all the tenants are in a healthy state.
  • Improved reliability of infrastructure backup operations during service failover.
  • A new version of the Azure Stack Nagios plugin is available that uses the Azure Active Directory authentication libraries (ADAL) for authentication. The plugin now also supports Azure AD and Active Directory Federation Services (AD FS) deployments of Azure Stack. For more information, see the Nagios plugin exchange site.
  • A new hybrid profile 2019-03-01-Hybrid was released that supports all the latest features in Azure Stack. Both Azure PowerShell and Azure CLI support the 2019-03-01-Hybrid profile. The .NET, Ruby, Node.js, Go, and Python SDKs have published packages that support the 2019-03-01-Hybrid profile. The respective documentation and some samples have been updated to reflect the changes.
  • The Node.js SDK now supports API profiles. Packages that support the 2019-03-01-Hybrid profile are published.
  • The 1905 Azure Stack update adds two new infrastructure roles to improve platform reliability and supportability:
    • Infrastructure ring: In the future, the infrastructure ring will host containerized versions of existing infrastructure roles – for example, xrp – that currently require their own designated infrastructure VMs. This will improve platform reliability and reduce the number of infrastructure VMs that Azure Stack requires. This subsequently reduces the overall resource consumption of Azure Stack’s infrastructure roles in the future.
    • Support ring: In the future, the support ring will be used to handle enhanced support scenarios for customers.In addition, we added an extra instance of the domain controller VM for improved availability of this role.These changes will increase the resource consumption of Azure Stack infrastructure in the following ways:Azure Stack SKUIncrease in Compute ConsumptionIncrease in Memory Consumption4 Nodes22 vCPU28 GB8 Nodes38 vCPU44 GB12 Nodes54 vCPU60 GB16 Nodes70 vCPU76 GB

Changes

  • To increase reliability and availability during planned and unplanned maintenance scenarios, Azure Stack adds an additional infrastructure role instance for domain services.
  • With this update, during repair and add node operations, the hardware is validated to ensure homogenous scale unit nodes within a scale unit.
  • If scheduled backups are failing to complete and the defined retention period is exceeded, the infrastructure backup controller will ensure at least one successful backup is retained.

Fixes

  • Fixed an issue in which a Compute host agent warning appeared after restarting a node in the scale unit.
  • Fixed issues in marketplace management in the administrator portal which showed incorrect results when filters were applied, and showed duplicate publisher names in the publisher filter. Also made performance improvements to display results faster.
  • Fixed issue in the available backup blade that listed a new available backup before it completed upload to the external storage location. Now the available backup will show in the list after it is successfully uploaded to the storage location.
  • Fixed issue with retrieving recovery keys during backup operation.
  • Fixed issue with OEM update displaying version as ‘undefined’ in operator portal.

Security updates

For information about security updates in this update of Azure Stack, see Azure Stack security updates.

Update planning

Before applying the update, make sure to review the following information:

Download the update

You can download the Azure Stack 1905 update package from the Azure Stack download page. When using the downloader tool, be sure to use the latest version and not a cached copy from your downloads directory.

Hotfixes

Azure Stack releases hotfixes on a regular basis. Be sure to install the latest Azure Stack hotfix for 1904 before updating Azure Stack to 1905.

Azure Stack hotfixes are only applicable to Azure Stack integrated systems; do not attempt to install hotfixes on the ASDK.

Before applying the 1905 update

The 1905 release of Azure Stack must be applied on the 1904 release with the following hotfixes:

After successfully applying the 1905 update

After the installation of this update, install any applicable hotfixes. For more information, see Microsofts’ servicing policy.

Automatic update notifications

Customers with systems that can access the internet from the infrastructure network will see the Update available message in the operator portal. Systems without internet access can download and import the .zip file with the corresponding .xml.

Tip
Subscribe to the following RSS or Atom feeds to keep up with Azure Stack hotfixes:

Known issues (post installation)

This article lists known issues in the 1905 release of Azure Stack. The list is updated as new issues are identified.

Important
Review this section before applying the update!

Update process

Host node update prerequisite failure

  • Applicable: This issue applies to the 1905 update.
  • Cause: When attempting to install the 1905 Azure Stack update, the status for the update might fail due to Host Node Update Prerequisite. This is generally caused by a host node having insufficient free disk space.
  • Remediation: Contact Azure Stack support to receive assistance clearing disk space on the host node.
  • Occurrence: Uncommon

Preparation failed

  • Applicable: This issue applies to all supported releases.
  • Cause: When attempting to install the 1905 Azure Stack update, the status for the update might fail and change state to PreparationFailed. This is caused by the update resource provider (URP) being unable to properly transfer the files from the storage container to an internal infrastructure share for processing. The 1905 update package is larger than previous update packages which may make this issue more likely to occur.
  • Remediation: Starting with version 1901 (1.1901.0.95), you can work around this issue by clicking Update now again (not Resume). The URP then cleans up the files from the previous attempt, and restarts the download. If the problem persists, we recommend manually uploading the update package by following the Import and install updates section.
  • Occurrence: Common

Portal

Subscription resources

  • Applicable: This issue applies to all supported releases.
  • Cause: Deleting user subscriptions results in orphaned resources.
  • Remediation: First delete user resources or the entire resource group, and then delete the user subscriptions.
  • Occurrence: Common

Subscription permissions

  • Applicable: This issue applies to all supported releases.
  • Cause: You cannot view permissions to your subscription using the Azure Stack portals.
  • Remediation: Use PowerShell to verify permissions.
  • Occurrence: Common

Marketplace management

  • Applicable: This issue applies to 1904 and 1905
  • Cause: The marketplace management screen is not visible when you sign in to the administrator portal.
  • Remediation: Refresh the browser or go to Settings and select the option Reset to default settings.
  • Occurrence: Intermittent

Docker extension

  • Applicable: This issue applies to all supported releases.
  • Cause: In both the administrator and user portals, if you search for Docker, the item is incorrectly returned. It is not available in Azure Stack. If you try to create it, an error is displayed.
  • Remediation: No mitigation.
  • Occurrence: Common

Upload blob

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, when you try to upload a blob using the OAuth(preview) option, the task fails with an error message.
  • Remediation: Upload the blob using the SAS option.
  • Occurrence: Common

Template

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, the template deployment UI does not populate parameters for the template names beginning with “_” (the underscore character).
  • Remediation: Remove the “_” (underscore character) from the template name.
  • Occurrence: Common

Networking

Load balancer

Add backend pool

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, if you attempt to add a Backend Pool to a Load Balancer, the operation fails with the error message failed to update Load Balancer….
  • Remediation: Use PowerShell, CLI or a Resource Manager template to associate the backend pool with a load balancer resource.
  • Occurrence: Common

Create inbound NAT

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, if you attempt to create an Inbound NAT Rule for a Load Balancer, the operation fails with the error message Failed to update Load Balancer….
  • Remediation: Use PowerShell, CLI or a Resource Manager template to associate the backend pool with a load balancer resource.
  • Occurrence: Common

Create load balancer

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, the Create Load Balancer window shows an option to create a Standard load balancer SKU. This option is not supported in Azure Stack.
  • Remediation: Use the Basic load balancer options instead.
  • Occurrence: Common

Public IP address

  • Applicable: This issue applies to all supported releases.
  • Cause: In the user portal, the Create Public IP Address window shows an option to create a Standard SKU. The Standard SKU is not supported in Azure Stack.
  • Remediation: Use the Basic SKU for public IP address.
  • Occurrence: Common

Compute

VM boot diagnostics

  • Applicable: This issue applies to all supported releases.
  • Cause: When creating a new Windows virtual machine (VM), the following error may be displayed: Failed to start virtual machine ‘vm-name’. Error: Failed to update serial output settings for VM ‘vm-name’. The error occurs if you enable boot diagnostics on a VM, but delete your boot diagnostics storage account.
  • Remediation: Recreate the storage account with the same name you used previously.
  • Occurrence: Common

VM resize

  • Applicable: This issue applies to the 1905 release.
  • Cause: Unable to successfully resize a managed disk VM. Attempting to resize the VM generates an error with “code”: “InternalOperationError”, “message”: “An internal error occurred in the operation.”
  • Remediation: We are working to remediate this in the next release. Currently, you must recreate the VM with the new VM size.
  • Occurrence: Common

Virtual machine scale set

CentOS

  • Applicable: This issue applies to all supported releases.
  • Cause: The virtual machine scale set creation experience provides CentOS-based 7.2 as an option for deployment. CentOS 7.2 is not available on Azure Stack Marketplace which will cause deployment failures calling out that the image is not found.
  • Remediation: Select another operating system for your deployment, or use an Azure Resource Manager template specifying another CentOS image that has been downloaded prior to deployment from the marketplace by the operator.
  • Occurrence: Common

Remove scale set

  • Applicable: This issue applies to all supported releases.
  • Cause: You cannot remove a scale set from the Virtual machine scale sets blade.
  • Remediation: Select the scale set that you want to remove, then click the Delete button from the Overview pane.
  • Occurrence: Common

Create failures during patch and update on 4-node Azure Stack environments

  • Applicable: This issue applies to all supported releases.
  • Cause: Creating VMs in an availability set of 3 fault domains and creating a virtual machine scale set instance fails with a FabricVmPlacementErrorUnsupportedFaultDomainSize error during the update process on a 4-node Azure Stack environment.
  • Remediation: You can create single VMs in an availability set with 2 fault domains successfully. However, scale set instance creation is still not available during the update process on a 4-node Azure Stack.

Scale set instance view blade doesn’t load

  • Applicable: This issue applies to 1904 and 1905 release.
  • Cause: The instance view blade of a virtual machine scale set located at Azure Stack portal -> Dashboard -> Virtual machine scale sets -> AnyScaleSet – Instances -> AnyScaleSetInstance fails to load, and displays a crying cloud image.
  • Remediation: There is currently no remediation and we are working on a fix. Until then, please use the CLI command az vmss get-instance-view to get the instance view of a scale set.

Ubuntu SSH access

  • Applicable: This issue applies to all supported releases.
  • Cause: An Ubuntu 18.04 VM created with SSH authorization enabled does not allow you to use the SSH keys to sign in.
  • Remediation: Use VM access for the Linux extension to implement SSH keys after provisioning, or use password-based authentication.
  • Occurrence: Common

Next steps

Posted on

Azure Stack 1904 Update

Applies to: Azure Stack integrated systems

This article describes the contents of the 1904 update package. The update includes what’s new improvements, and fixes for this release of Azure Stack. This article contains the following information:

Important!

This update package is only for Azure Stack integrated systems. Do not apply this update package to the Azure Stack Development Kit.

Build reference

The Azure Stack 1904 update build number is 1.1904.0.36.

What’s in this update

Improvements

  • Significant improvements have been made to the Software Defined Networking (SDN) Stack in 1904. These improvements increase the overall servicing and reliability of the SDN stack in Azure Stack.
  • Added a notification in the administrator portal, when the currently logged in user does not have the necessary permissions, which enables the dashboard to load properly. It also contains a link to the documentation that explains which accounts have the appropriate permissions, depending on the identity provider used during deployment.
  • Added improvements to VM resiliency and uptime, which resolves the scenario in which all VMs go offline if the storage volume containing the VM configuration files goes offline.
  • Added optimization to the number of VMs evacuated concurrently and placed a cap on bandwidth consumed, to address VM brownouts or blackouts if the network is under heavy load. This change increases VM uptime when a system is updating.
  • Improved resource throttling when a system is running at scale to protect against internal processes exhausting platform resources, resulting in failed operations in the portal.
  • Improved filtering capabilities enable operators to apply multiple filters at the same time. You can only sort on the Namecolumn in the new user interface.
  • Improvements to the process of deleting offers, plans, quotas, and subscriptions. You can now successfully delete offers, quotas, plans, and subscriptions from the Administrator portal if the object you want to delete has no dependencies. For more information, see this article.
  • Improved syslog message volume by filtering out unnecessary events and providing a configuration parameter to select desired severity level for forwarded messages. For more information about how to configure the severity level, see Azure Stack datacenter integration – syslog forwarding.
  • The Azure Stack Infrastructure consumes an additional 12 GB + (4 GB * Number of Azure Stack hosts) from the 1904 update onwards. This means that in a 4 node stamp there will be an additional capacity consumption of 28 GB (12 GB + 4 GB * 4) reflected in the capacity screen of the Azure Stack administrator portal. Your update to the 1904 release should succeed even if the additional memory consumption puts your Azure Stack stamp over capacity. If your Azure Stack stamp is over memory usage AFTER the update is completed, you will see an alert reflecting this state, with remediation steps to de-allocate some VMs.
  • Added a new capability to the Get-AzureStackLog cmdlet by incorporating an additional parameter, -OutputSASUri. You can now collect Azure Stack logs from your environment and store them in the specified Azure Storage blob container. For more information, see Azure Stack diagnostics.
  • Added a new memory check in the Test-AzureStack UpdateReadiness group, which checks to see if you have enough memory available on the stack for the update to complete successfully.
  • Improvements to Test-AzureStack for evaluating Service Fabric health.
  • Improvements to hardware updates, which reduces the time it takes to complete drive firmware update to 2-4 hours. The update engine dynamically determines which portions of the update need to execute, based on content in the package.
  • Added robust operation prechecks to prevent disruptive infrastructure role instance operations that affect availability.
  • Improvements to idempotency of infrastructure backup action plan.
  • Improvements to Azure Stack log collection. These improvements reduce the time it takes to retrieve the set of logs. Also, the Get-AzureStackLog cmdlet no longer generates default logs for the OEM role. You must execute the Invoke-AzureStackOnDemandLog cmdlet, specifying the role to retrieve the OEM logs. For more information , see Azure Stack diagnostics.
  • Azure Stack now monitors the federation data URL provided for datacenter integration with ADFS. This improves reliability during secret rotation of the customer ADFS instance or farm.

Changes

  • Removed the option for Azure Stack operators to shut down infrastructure role instances in the administrator portal. The restart functionality ensures a clean shutdown attempt before restarting the infrastructure role instance. For advanced scenarios, the API and PowerShell functionality remains available.
  • There is a new Marketplace management experience, with separate screens for Marketplace images and resource providers. For now, the Resource providers window is empty, but in future releases new PaaS service offerings will appear and be managed in the Resource providers window.
  • Changes to the update experience in the operator portal. There is a new grid for resource provider updates. The ability to update resource providers is not available yet.
  • Changes to the update installation experience in the operator portal. To help Azure Stack operators respond appropriately to an update issue, the portal now provides more specific recommendations based on the health of the scale unit, as derived automatically by running Test-AzureStack and parsing the results. Based on the result, it will inform the operator to take one of two actions:
    • A “soft” warning alert is displayed in the portal that reads “The most recent update needs attention. Microsoft recommends opening a service request during normal business hours. As part of the update process, Test-AzureStack is performed, and based on the output we generate the most appropriate alert. In this case, Test-AzureStack passed.”
    • A “hard” critical alert is displayed in the portal that reads, “The most recent update failed. Microsoft recommends opening a service request as soon as possible. As part of the update process, Test-AzureStack is performed, and based on the output we generate the most appropriate alert. In this case, Test-AzureStack also failed.”
  • Updated Azure Linux Agent version 2.2.38.0. This support allows customers to maintain consistent Linux images between Azure and Azure Stack.

Fixes

  • Fixed an issue in which the syslog configuration was not persisted through an update cycle, causing the syslog client to lose its configuration, and the syslog messages to stop being forwarded. Syslog configuration is now preserved.
  • Fixed an issue in CRP that blocked deallocation of VMs. Previously, if a VM contained multiple large managed disks, deallocating the VM might have failed with a timeout error.
  • Fixed issue with Windows Defender engine impacting access to scale-unit storage.
  • Fixed a user portal issue in which the Access Policy window for blob storage accounts failed to load.
  • Fixed an issue in both administrator and user portals, in which erroneous notifications about the global Azure portal were displayed.
  • Fixed a user portal issue in which selecting the Feedback tile caused an empty browser tab to open.
  • Fixed a portal issue in which changing a static IP address for an IP configuration that was bound to a network adapter attached to a VM instance, caused an error message to be displayed.
  • Fixed a user portal issue in which attempting to Attach Network Interface to an existing VM via the Networking window caused the operation to fail with an error message.
  • Fixed an issue in which Azure Stack did not support attaching more than 4 Network Interfaces (NICs) to a VM instance.
  • Fixed a portal issue in which adding an inbound security rule and selecting Service Tag as the source, displayed several options that are not available for Azure Stack.
  • Fixed the issue in which Network Security Groups (NSGs) did not work in Azure Stack in the same way as global Azure.
  • Fixed an issue in Marketplace management, which hides all downloaded products if registration expires or is removed.
  • Fixed an issue in which issuing a Set-AzureRmVirtualNetworkGatewayConnection command in PowerShell to an existing virtual network gateway connection failed with the error message Invalid shared key configured….
  • Fixed an issue that caused the Network Resource Provider (NRP) to be out of sync with the network controller, resulting in duplicate resources being requested. In some cases, this resulted in leaving the parent resource in an error state.
  • Fixed an issue in which if a user that was assigned a contributor role to a subscription, but was not explicitly given read permissions, an error was generated that read …The client ‘somelogonaccount@domain.com’ with object ID {GUID} does not have authorization to perform action… when attempting to save a change to a resource.
  • Fixed an issue in which the marketplace management screen was empty if the offline syndication tool was used to upload images, and any one of them was missing the icon URI(s).

Security updates

This update of Azure Stack does not include security updates to the underlying operating system which hosts Azure Stack. For information, see Azure Stack security updates.

Update planning

Before applying the update, make sure to review the following information:

Note!
Make sure to use the latest version of the Azure Stack Capacity Planner tool to perform your workload planning and sizing. The latest version contains bug fixes and provides new features that are released with each Azure Stack update.

Download the update

You can download the Azure Stack 1904 update package from the Azure Stack download page.

Hotfixes

Azure Stack releases hotfixes on a regular basis. Be sure to install the latest Azure Stack hotfix for 1903 before updating Azure Stack to 1904.

Azure Stack hotfixes are only applicable to Azure Stack integrated systems; do not attempt to install hotfixes on the ASDK.

Before applying the 1904 update

The 1904 release of Azure Stack must be applied on the 1903 release with the following hotfixes!

After successfully applying the 1904 update

After the installation of this update, install any applicable hotfixes. For more information, see our Servicing Policy.

  • No hotfixes available for 1904.

Automatic update notifications

Customers with systems that can access the internet from the infrastructure network will see the Update available message in the operator portal. Systems without internet access can download and import the .zip file with the corresponding .xml.

 Tip

Subscribe to the following RSS or Atom feeds to keep up with Azure Stack hotfixes:

Next steps