Setup

Edit This Page

Adding Windows nodes and scheduling Windows containers in Kubernetes

Windows applications constitute a large portion of the services and applications that run in many organizations. Windows containers provide a modern way to encapsulate processes and package dependencies, making it easier to use DevOps practices and follow cloud native patterns for Windows applications. Kubernetes has become the defacto standard container orchestrator, and the release of Kubernetes 1.14 includes production support for scheduling Windows containers on Windows nodes in a Kubernetes cluster, enabling a vast ecosystem of Windows applications to leverage the power of Kubernetes. Organizations with investments in Windows-based applications and Linux-based applications don’t have to look for separate orchestrators to manage their workloads, leading to increased operational efficiencies across their deployments, regardless of operating system.

Intro to Windows containers in Kubernetes

To enable the orchestration of Windows containers in Kubernetes, simply include Windows nodes in your existing Linux cluster. Scheduling Windows containers in Pods on Kubernetes is as simple and easy as scheduling Linux-based containers.

In order to run Windows containers, your Kubernetes cluster must include multiple operating systems, with control plane nodes running Linux and workers running either Windows or Linux depending on your workload needs. Windows Server 2019 is the only Windows operating system supported, enabling Kubernetes Node on Windows (including kubelet, container runtime, and kube-proxy). For a detailed explanation of Windows distribution channels see the Microsoft documentation.

Note: The Kubernetes control plane, including the master components, will continue to run on Linux. There are no plans to have a Windows-only Kubernetes cluster.
Note: In this document, when we talk about Windows containers we mean Windows containers with process isolation. Windows containers with Hyper-V isolation is planned for a future release.

Supported Functionality and Limitations

Supported Functionality

Compute

From an API and kubectl perspective, Windows containers behave in much the same way as Linux-based containers. However, there are some notable differences in key functionality which are outlined in the limitation section.

Let’s start with the operating system version. Refer to the following table for Windows operating system support in Kubernetes. A single heterogeneous Kubernetes cluster can have both Windows and Linux worker nodes. Windows containers have to be scheduled on Windows nodes and Linux containers on Linux nodes.

Kubernetes version Host OS version (Kubernetes Node)
Windows Server 1709 Windows Server 1803 Windows Server 1809/Windows Server 2019
Kubernetes v1.14 Not Supported Not Supported Supported for Windows Server containers Builds 17763.* with Docker EE-basic 18.09
Note: The Windows Server Host Operating System is subject to the Windows Server licensing. The Windows Container images are subject to the Supplemental License Terms for Windows containers.
Note: Windows containers with process isolation have strict compatibility rules, where the host OS version must match the container base image OS version. Once we support Windows containers with Hyper-V isolation in Kubernetes, the limitation and compatibility rules will change.

Key Kubernetes elements work the same way in Windows as they do in Linux. In this section, we will talk about some of the key workload enablers and how they map to Windows.

Pods, Controllers and Services are critical elements to managing Windows workloads on Kubernetes. However, on their own they are not enough to enable the proper lifecycle management of Windows workloads in a dynamic cloud native environment. We added support for the following features:

Container Runtime

Docker EE-basic 18.09 is required on Windows Server 2019 / 1809 nodes for Kubernetes. This works with the dockershim code included in the kubelet. Additional runtimes such as CRI-ContainerD may be supported in later Kubernetes versions.

Storage

Kubernetes Volumes enable complex applications with data persistence and Pod volume sharing requirements to be deployed on Kubernetes. Kubernetes on Windows supports the following types of volumes:

Networking

Networking for Windows containers is exposed through CNI plugins. Windows containers function similarly to virtual machines in regards to networking. Each container has a virtual network adapter (vNIC) which is connected to a Hyper-V virtual switch (vSwitch). The Host Networking Service (HNS) and the Host Compute Service (HCS) work together to create containers and attach container vNICs to networks. HCS is responsible for the management of containers whereas HNS is responsible for the management of networking resources such as:

The following service spec types are supported:

Windows supports five different networking drivers/modes: L2bridge, L2tunnel, Overlay, Transparent, and NAT. In a heterogeneous cluster with Windows and Linux worker nodes, you need to select a networking solution that is compatible on both Windows and Linux. The following out-of-tree plugins are supported on Windows, with recommendations on when to use each CNI:

Network Driver Description Container Packet Modifications Network Plugins Network Plugin Characteristics
L2bridge Containers are attached to an external vSwitch. Containers are attached to the underlay network, although the physical network doesn't need to learn the container MACs because they are rewritten on ingress/egress. Inter-container traffic is bridged inside the container host. MAC is rewritten to host MAC, IP remains the same. win-bridge, Azure-CNI, Flannel host-gateway uses win-bridge win-bridge uses L2bridge network mode, connects containers to the underlay of hosts, offering best performance. Requires L2 adjacency between container hosts
L2Tunnel This is a special case of l2bridge, but only used on Azure. All packets are sent to the virtualization host where SDN policy is applied. MAC rewritten, IP visible on the underlay network Azure-CNI Azure-CNI allows integration of containers with Azure vNET, and allows them to leverage the set of capabilities that Azure Virtual Network provides. For example, securely connect to Azure services or use Azure NSGs. See azure-cni for some examples
Overlay (Overlay networking for Windows in Kubernetes is in alpha stage) Containers are given a vNIC connected to an external vSwitch. Each overlay network gets its own IP subnet, defined by a custom IP prefix.The overlay network driver uses VXLAN encapsulation. Encapsulated with an outer header, inner packet remains the same. Win-overlay, Flannel VXLAN (uses win-overlay) win-overlay should be used when virtual container networks are desired to be isolated from underlay of hosts (e.g. for security reasons). Allows for IPs to be re-used for different overlay networks (which have different VNID tags) if you are restricted on IPs in your datacenter. This option may be used when the container hosts are not L2 adjacent but have L3 connectivity
Transparent (special use case for ovn-kubernetes) Requires an external vSwitch. Containers are attached to an external vSwitch which will enable intra-pod communication via logical networks (logical switches and routers). Packet is encapsulated either via GENEVE or STT tunneling to reach pods which are not on the same host. Packets are forwarded or dropped via the tunnel metadata information supplied by the ovn network controller. NAT is done for north-south communication. ovn-kubernetes Deploy via ansible. Distributed ACLs can be applied via Kubernetes policies. IPAM support. Load-balancing can be achieved without kube-proxy. NATing is done without using iptables/netsh.
NAT (not used in Kubernetes) Containers are given a vNIC connected to an internal vSwitch. DNS/DHCP is provided using an internal component called WinNAT MAC and IP is rewritten to host MAC/IP. nat Included here for completeness

As outlined above, the Flannel CNI meta plugin is also supported on Windows via the VXLAN network backend (alpha support ; delegates to win-overlay) and host-gateway network backend (stable support; delegates to win-bridge). This plugin supports delegating to one of the reference CNI plugins (win-overlay, win-bridge), to work in conjunction with Flannel daemon on Windows (Flanneld) for automatic node subnet lease assignment and HNS network creation. This plugin reads in its own configuration file (net-conf.json), and aggregates it with the environment variables from the FlannelD generated subnet.env file. It then delegates to one of the reference CNI plugins for network plumbing, and sends the correct configuration containing the node-assigned subnet to the IPAM plugin (e.g. host-local).

For the node, pod, and service objects, the following network flows are supported for TCP/UDP traffic:

The following IPAM options are supported on Windows:

Limitations

Control Plane

Windows is only supported as a worker node in the Kubernetes architecture and component matrix. This means that a Kubernetes cluster must always include Linux master nodes, zero or more Linux worker nodes, and zero or more Windows worker nodes.

Compute

Resource management and process isolation

Linux cgroups are used as a pod boundary for resource controls in Linux. Containers are created within that boundary for network, process and file system isolation. The cgroups APIs can be used to gather cpu/io/memory stats. In contrast, Windows uses a Job object per container with a system namespace filter to contain all processes in a container and provide logical isolation from the host. There is no way to run a Windows container without the namespace filtering in place. This means that system privileges cannot be asserted in the context of the host, and thus privileged containers are not available on Windows. Containers cannot assume an identity from the host because the Security Account Manager (SAM) is separate.

Operating System Restrictions

Windows has strict compatibility rules, where the host OS version must match the container base image OS version. Only Windows containers with a container operating system of Windows Server 2019 are supported. Hyper-V isolation of containers, enabling some backward compatibility of Windows container image versions, is planned for a future release.

Feature Restrictions
Memory Reservations and Handling

Windows does not have an out-of-memory process killer as Linux does. Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. The net effect is that Windows won’t reach out of memory conditions the same way Linux does, and processes will page to disk instead of being subject to out of memory (OOM) termination. If memory is over-provisioned and all physical memory is exhausted, then paging can slow down performance.

Keeping memory usage within reasonable bounds is possible with a two-step process. First, use the kubelet parameters --kubelet-reserve and/or --system-reserve to account for memory usage on the node (outside of containers). This will reduce NodeAllocatable). As you deploy workloads, use resource limits (must set only limits or limits must equal requests) on containers. This will also subtract from NodeAllocatable and prevent the scheduler from adding more pods once a node is full.

A best practice to avoid over-provisioning is to configure the kubelet with a system reserved memory of at least 2GB to account for Windows, Docker, and Kubernetes processes.

The behavior of the flags behave differently as described below:

Storage

Windows has a layered filesystem driver to mount container layers and create a copy filesystem based on NTFS. All file paths in the container are resolved only within the context of that container.

As a result, the following storage functionality is not supported on Windows nodes

Networking

Windows Container Networking differs in some important ways from Linux networking. The Microsoft documentation for Windows Container Networking contains additional details and background.

The Windows host networking networking service and virtual switch implement namespacing and can create virtual NICs as needed for a pod or container. However, many configurations such as DNS, routes, and metrics are stored in the Windows registry database rather than /etc/… files as they are on Linux. The Windows registry for the container is separate from that of the host, so concepts like mapping /etc/resolv.conf from the host into a container don’t have the same effect they would on Linux. These must be configured using Windows APIs run in the context of that container. Therefore CNI implementations need to call the HNS instead of relying on file mappings to pass network details into the pod or container.

The following networking functionality is not supported on Windows nodes

CNI Plugins
  1. Node-pod connectivity isn’t possible by design. It’s only possible for local pods with Flannel PR 1096
  2. We are restricted to using VNI 4096 and UDP port 4789. The VNI limitation is being worked on and will be overcome (open-source flannel changes). See official Flannel VXLAN backend docs for more details on these parameters.
DNS
Security

Secrets are written in clear text on the node’s volume (as compared to tmpfs/in-memory on linux). This means customers have to do two things

  1. Use file ACLs to secure the secrets file location
  2. Use volume-level encryption using BitLocker

RunAsUser is not currently supported on Windows. The workaround is to create local accounts before packaging the container. The RunAsUsername capability may be added in a future release.

Linux specific pod security context privileges such as SELinux, AppArmor, Seccomp, Capabilities (POSIX Capabilities), and others are not supported.

In addition, as mentioned already, privileged containers are not supported on Windows.

API

There are no differences in how most of the Kubernetes APIs work for Windows. The subtleties around what’s different come down to differences in the OS and container runtime. In certain situations, some properties on workload APIs such as Pod or Container were designed with an assumption that they are implemented on Linux, failing to run on Windows.

At a high level, these OS concepts are different:

Exit Codes follow the same convention where 0 is success, nonzero is failure. The specific error codes may differ across Windows and Linux. However, exit codes passed from the Kubernetes components (kubelet, kube-proxy) will be unchanged.

V1.Container
V1.Pod
V1.PodSecurityContext

None of the PodSecurityContext fields work on Windows. They’re listed here for reference.

User Guide: Add Windows Nodes in Kubernetes

Objectives

The Kubernetes platform can now be used to run both Linux and Windows containers. One or more Windows nodes can be registered to a cluster. This guide shows how to:

Before you begin

Getting Started: Adding a Windows Node to Your Cluster

Plan IP Addressing

Kubernetes cluster management requires careful planning of your IP addresses so that you do not inadvertently cause network collision. This guide assumes that you are familiar with the Kubernetes networking concepts.

In order to deploy your cluster you will need the following address spaces:

Subnet / address range Description Default value
--- --- ---
Service Subnet A non-routable, purely virtual subnet that is used by pods to uniformly access services without caring about the network topology. It is translated to/from routable address space by kube-proxy running on the nodes. "10.96.0.0/12"
Cluster Subnet This is a global subnet that is used by all pods in the cluster. Each node is assigned a smaller /24 subnet from this for their pods to use. It must be large enough to accommodate all pods used in your cluster. To calculate minimumsubnet size: (number of nodes) + (number of nodes * maximum pods per node that you configure) Example: for a 5 node cluster for 100 pods per node: (5) + (5 * 100) = 505. "10.244.0.0/16"
Kubernetes DNS Service IP IP address of kube-dns service that will be used for DNS resolution & cluster service discovery. "10.96.0.10"

Review the networking options supported in ‘Intro to Windows containers in Kubernetes: Supported Functionality: Networking’ to determine how you need to allocate IP addresses for your cluster.

Components that run on Windows

While the Kubernetes control plane runs on your Linux node(s), the following components will be configured and run on your Windows node(s).

  1. kubelet
  2. kube-proxy
  3. kubectl (optional)
  4. Container runtime

Get the latest binaries from https://github.com/kubernetes/kubernetes/releases, starting with v1.14 or later. The Windows-amd64 binaries for kubeadm, kubectl, kubelet, and kube-proxy can be found under the CHANGELOG link.

Networking Configuration

Once you have a Linux-based Kubernetes master node you are ready to choose a networking solution. This guide illustrates using Flannel in VXLAN mode for simplicity.

Configuring Flannel in VXLAN mode on the Linux controller
  1. Prepare Kubernetes master for Flannel

    Some minor preparation is recommended on the Kubernetes master in our cluster. It is recommended to enable bridged IPv4 traffic to iptables chains when using Flannel. This can be done using the following command:

    sudo sysctl net.bridge.bridge-nf-call-iptables=1
  2. Download & configure Flannel

    Download the most recent Flannel manifest:

    wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

    There are two sections you should modify to enable the vxlan networking backend:

    After applying the steps below, the net-conf.json section of kube-flannel.yml should look as follows:

    net-conf.json: |
        {
          "Network": "10.244.0.0/16",
          "Backend": {
            "Type": "vxlan",
            "VNI" : 4096,
            "Port": 4789
          }
        }
Note: The VNI must be set to 4096 and port 4789 for Flannel on Linux to interoperate with Flannel on Windows. Support for other VNIs is coming soon. See VXLAN for an explanation of these fields.
  1. In the net-conf.json section of your kube-flannel.yml, double-check:

    1. The cluster subnet (e.g. “10.244.0.0/16”) is set as per your IP plan.
      • VNI 4096 is set in the backend
      • Port 4789 is set in the backend
    2. In the cni-conf.json section of your kube-flannel.yml, change the network name to vxlan0.

    Your cni-conf.json should look as follows:

    cni-conf.json: |
        {
          "name": "vxlan0",
          "plugins": [
            {
              "type": "flannel",
              "delegate": {
                "hairpinMode": true,
                "isDefaultGateway": true
              }
            },
            {
              "type": "portmap",
              "capabilities": {
                "portMappings": true
              }
            }
          ]
        }
  2. Apply the Flannel yaml and Validate

    Let’s apply the Flannel configuration:

    kubectl apply -f kube-flannel.yml

    Next, since the Flannel pods are Linux-based, apply a NodeSelector patch, which can be found here, to the Flannel DaemonSet pod:

    kubectl patch ds/kube-flannel-ds-amd64 --patch "$(cat node-selector-patch.yml)" -n=kube-system

    After a few minutes, you should see all the pods as running if the Flannel pod network was deployed.

    kubectl get pods --all-namespaces

    alt_text

    Verify that the Flannel DaemonSet has the NodeSelector applied.

    kubectl get ds -n kube-system

    alt_text

Join Windows Worker

In this section we’ll cover configuring a Windows node from scratch to join a cluster on-prem. If your cluster is on a cloud you’ll likely want to follow the cloud specific guides in the next section.

Preparing a Windows Node
Note: All code snippets in Windows sections are to be run in a PowerShell environment with elevated permissions (Admin).
  1. Install Docker (requires a system reboot)

    Kubernetes uses Docker as its container engine, so we need to install it. You can follow the official Docs instructions, the Docker instructions, or try the following recommended steps:

    Enable-WindowsOptionalFeature -FeatureName Containers
    Restart-Computer -Force
    Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
    Install-Package -Name Docker -ProviderName DockerMsftProvider

    If you are behind a proxy, the following PowerShell environment variables must be defined:

    [Environment]::SetEnvironmentVariable("HTTP_PROXY", "http://proxy.example.com:80/", [EnvironmentVariableTarget]::Machine)
    [Environment]::SetEnvironmentVariable("HTTPS_PROXY", "http://proxy.example.com:443/", [EnvironmentVariableTarget]::Machine)

    If after reboot you see the following error, you need to restart the docker service manually

    alt_text

    Start-Service docker
Note: The “pause” (infrastructure) image is hosted on Microsoft Container Registry (MCR). You can access it using docker pull mcr.microsoft.com/k8s/core/pause:1.0.0. The DOCKERFILE is available at https://github.com/Microsoft/SDN/blob/master/Kubernetes/windows/Dockerfile.
  1. Prepare a Windows directory for Kubernetes

    Create a “Kubernetes for Windows” directory to store Kubernetes binaries as well as any deployment scripts and config files.

    mkdir c:\k
  2. Copy Kubernetes certificate

    Copy the Kubernetes certificate file $HOME/.kube/config from the Linux controller to this new C:\k directory on your Windows node.

    Tip: You can use tools such as xcopy, WinSCP, or this PowerShell wrapper for WinSCP to transfer the config file between nodes.

  3. Download Kubernetes binaries

    To be able to run Kubernetes, you first need to download the kubelet and kube-proxy binaries. You download these from the Node Binaries links in the CHANGELOG.md file of the latest releases. For example ‘kubernetes-node-windows-amd64.tar.gz’. You may also optionally download kubectl to run on Windows which you can find under Client Binaries.

    Use the Expand-Archive PowerShell command to extract the archive and place the binaries into C:\k.

Join the Windows node to the Flannel cluster

The Flannel overlay deployment scripts and documentation are available in this repository. The following steps are a simple walkthrough of the more comprehensive instructions available there.

Download the Flannel start.ps1 script, the contents of which should be extracted to C:\k:

cd c:\k
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
wget https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/flannel/start.ps1 -o c:\k\start.ps1
Note: start.ps1 references install.ps1, which will download additional files such as the flanneld executable and the Dockerfile for infrastructure pod and install those for you. For overlay networking mode, the firewall will be opened for local UDP port 4789. There may be multiple powershell windows being opened/closed as well as a few seconds of network outage while the new external vSwitch for the pod network is being created the first time. Run the script using the arguments as specified below:
.\start.ps1 -ManagementIP <Windows Node IP> -NetworkMode overlay  -ClusterCIDR <Cluster CIDR> -ServiceCIDR <Service CIDR> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Log directory>
Parameter Default Value Notes
--- --- ---
-ManagementIP N/A (required) The IP address assigned to the Windows node. You can use ipconfig to find this.
-NetworkMode l2bridge We're using overlay here
-ClusterCIDR 10.244.0.0/16 Refer to your cluster IP plan
-ServiceCIDR 10.96.0.0/12 Refer to your cluster IP plan
-KubeDnsServiceIP 10.96.0.10
-InterfaceName Ethernet The name of the network interface of the Windows host. You can use ipconfig to find this.
-LogDir C:\k The directory where kubelet and kube-proxy logs are redirected into their respective output files.

Now you can view the Windows nodes in your cluster by running the following:

kubectl get nodes
Note: You may want to configure your Windows node components like kubelet and kube-proxy to run as services. View the services and background processes section under troubleshooting for additional instructions. Once you are running the node components as services, collecting logs becomes an important part of troubleshooting. View the gathering logs section of the contributing guide for further instructions.

Public Cloud Providers

Azure

AKS-Engine can deploy a complete, customizable Kubernetes cluster with both Linux & Windows nodes. There is a step-by-step walkthrough available in the docs on GitHub.

GCP

Users can easily deploy a complete Kubernetes cluster on GCE following this step-by-step walkthrough on GitHub

Deployment with kubeadm and cluster API

Kubeadm is becoming the de facto standard for users to deploy a Kubernetes cluster. Windows node support in kubeadm will come in a future release. We are also making investments in cluster API to ensure Windows nodes are properly provisioned.

Next Steps

Now that you’ve configured a Windows worker in your cluster to run Windows containers you may want to add one or more Linux nodes as well to run Linux containers. Now you’re ready to proceed to the next step to schedule Windows containers on your cluster.

User Guide: Scheduling Windows containers in Kubernetes

Objectives

Before you begin

Getting Started: Deploying a Windows container

To deploy a Windows container on Kubernetes, you must first create an example application. The example YAML file below creates a simple webserver application. Create a service spec named win-webserver.yaml with the contents below:

    apiVersion: v1
    kind: Service
    metadata:
      name: win-webserver
      labels:
        app: win-webserver
    spec:
      ports:
      # the port that this service should serve on
      - port: 80
        targetPort: 80
      selector:
        app: win-webserver
      type: NodePort
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      labels:
        app: win-webserver
      name: win-webserver
    spec:
      replicas: 2
      template:
        metadata:
          labels:
            app: win-webserver
          name: win-webserver
        spec:
          containers:
          - name: windowswebserver
            image: mcr.microsoft.com/windows/servercore:ltsc2019
            command:
            - powershell.exe
            - -command
            - "<#code used from https://gist.github.com/wagnerandrade/5424431#> ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('> {0}' -f $$requestUrl) ;  ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='<html><body><H1>Windows Container Web Server</H1>' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='<p>IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='</body></html>' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('< {0}' -f $$responseStatus)  } ; "
          nodeSelector:
            beta.kubernetes.io/os: windows
Note: Port mapping is also supported, but for simplicity in this example the container port 80 is exposed directly to the service.
  1. Check that all nodes are healthy:

    kubectl get nodes
  2. Deploy the service and watch for pod updates:

    kubectl apply -f win-webserver.yaml
    kubectl get pods -o wide -w

    When the service is deployed correctly both Pods will be marked as Ready. To exit the watch command, press Ctrl+C.

  3. Check that the deployment succeeded. To verify:

    • Two containers per pod on the Windows node, use docker ps
    • Two pods listed from the Linux master, use kubectl get pods
    • Node-to-pod communication across the network, curl port 80 of your pod IPs from the Linux master to check for a web server response
    • Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node) using docker exec or kubectl exec
    • Service-to-pod communication, curl the virtual service IP (seen under kubectl get services) from the Linux master and from individual pods
    • Service discovery, curl the service name with the Kubernetes default DNS suffix
    • Inbound connectivity, curl the NodePort from the Linux master or machines outside of the cluster
    • Outbound connectivity, curl external IPs from inside the pod using kubectl exec
Note: Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack. Only Windows pods are able to access service IPs.

Managing Workload Identity with Group Managed Service Accounts

Starting with Kubernetes v1.14, Windows container workloads can be configured to use Group Managed Service Accounts (GMSA). Group Managed Service Accounts are a specific type of Active Directory account that provides automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers. Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA. Learn more about configuring and using GMSA for Windows containers here.

Taints and Tolerations

Users today will need to use some combination of taints and node selectors in order to keep Linux and Windows workloads on their respective OS-specific nodes. This will likely impose a burden only on Windows users. The recommended approach is outlined below, with one of its main goals being that this approach should not break compatibility for existing Linux workloads.

Ensuring OS-specific workloads land on the appropriate container host

Users can ensure Windows containers can be scheduled on the appropriate host using Taints and Tolerations. All Kubernetes nodes today have the following default labels:

If a Pod specification does not specify a nodeSelector like "beta.kubernetes.io/os": windows, it is possible the Pod can be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice is to use a nodeSelector.

However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers, as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators. In those situations, you may be hesitant to make the configuration change to add nodeSelectors. The alternative is to use Taints. Because the kubelet can set Taints during registration, it could easily be modified to automatically add a taint when running on Windows only.

For example: --register-with-taints='os=Win1809:NoSchedule'

By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). In order for a Windows Pod to be scheduled on a Windows node, it would need both the nodeSelector to choose Windows, and the appropriate matching toleration.

nodeSelector:
    "beta.kubernetes.io/os": windows
tolerations:
    - key: "os"
      operator: "Equal"
      value: "Win1809"
      effect: "NoSchedule"

Getting Help and Troubleshooting

Your main source of help for troubleshooting your Kubernetes cluster should start with this section. Some additional, Windows-specific troubleshooting help is included in this section. Logs are an important element of troubleshooting issues in Kubernetes. Make sure to include them any time you seek troubleshooting assistance from other contributors. Follow the instructions in the SIG-Windows contributing guide on gathering logs.

  1. How do I know start.ps1 completed successfully?

    You should see kubelet, kube-proxy, and (if you chose Flannel as your networking solution) flanneld host-agent processes running on your node, with running logs being displayed in separate PowerShell windows. In addition to this, your Windows node should be listed as “Ready” in your Kubernetes cluster.

  2. Can I configure the Kubernetes node processes to run in the background as services?

    Kubelet and kube-proxy are already configured to run as native Windows Services, offering resiliency by re-starting the services automatically in the event of failure (for example a process crash). You have two options for configuring these node components as services.

    1. As native Windows Services

      Kubelet & kube-proxy can be run as native Windows Services using sc.exe.

      # Create the services for kubelet and kube-proxy in two separate commands
      sc.exe create <component_name> binPath= "<path_to_binary> --service <other_args>"
      
      # Please note that if the arguments contain spaces, they must be escaped.
      sc.exe create kubelet binPath= "C:\kubelet.exe --service --hostname-override 'minion' <other_args>"
      
      # Start the services
      Start-Service kubelet
      Start-Service kube-proxy
      
      # Stop the service
      Stop-Service kubelet (-Force)
      Stop-Service kube-proxy (-Force)
      
      # Query the service status
      Get-Service kubelet
      Get-Service kube-proxy
    2. Using nssm.exe

      You can also always use alternative service managers like nssm.exe to run these processes (flanneld, kubelet & kube-proxy) in the background for you. You can use this sample script, leveraging nssm.exe to register kubelet, kube-proxy, and flanneld.exe to run as Windows services in the background.

      register-svc.ps1 -NetworkMode <Network mode> -ManagementIP <Windows Node IP> -ClusterCIDR <Cluster subnet> -KubeDnsServiceIP <Kube-dns Service IP> -LogDir <Directory to place logs>
              
      # NetworkMode      = The network mode l2bridge (flannel host-gw, also the default value) or overlay (flannel vxlan) chosen as a network solution
      # ManagementIP     = The IP address assigned to the Windows node. You can use ipconfig to find this
      # ClusterCIDR      = The cluster subnet range. (Default value 10.244.0.0/16)
      # KubeDnsServiceIP = The Kubernetes DNS service IP (Default value 10.96.0.10)
      # LogDir           = The directory where kubelet and kube-proxy logs are redirected into their respective output files (Default value C:\k)

      If the above referenced script is not suitable, you can manually configure nssm.exe using the following examples.

      # Register flanneld.exe
      nssm install flanneld C:\flannel\flanneld.exe
      nssm set flanneld AppParameters --kubeconfig-file=c:\k\config --iface=<ManagementIP> --ip-masq=1 --kube-subnet-mgr=1
      nssm set flanneld AppEnvironmentExtra NODE_NAME=<hostname>
      nssm set flanneld AppDirectory C:\flannel 
      nssm start flanneld
              
      # Register kubelet.exe
      nssm install kubelet C:\k\kubelet.exe
      nssm set kubelet AppParameters --hostname-override=<hostname> --v=6 --pod-infra-container-image=kubeletwin/pause --resolv-conf="" --allow-privileged=true --enable-debugging-handlers --cluster-dns=<DNS-service-IP> --cluster-domain=cluster.local --kubeconfig=c:\k\config --hairpin-mode=promiscuous-bridge --image-pull-progress-deadline=20m --cgroups-per-qos=false  --log-dir=<log directory> --logtostderr=false --enforce-node-allocatable="" --network-plugin=cni --cni-bin-dir=c:\k\cni --cni-conf-dir=c:\k\cni\config
      nssm set kubelet AppDirectory C:\k
      nssm start kubelet
        
      # Register kube-proxy.exe (l2bridge / host-gw)
      nssm install kube-proxy C:\k\kube-proxy.exe
      nssm set kube-proxy AppDirectory c:\k
      nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --hostname-override=<hostname>--kubeconfig=c:\k\config --enable-dsr=false --log-dir=<log directory> --logtostderr=false
      nssm.exe set kube-proxy AppEnvironmentExtra KUBE_NETWORK=cbr0
      nssm set kube-proxy DependOnService kubelet
      nssm start kube-proxy
        
      # Register kube-proxy.exe (overlay / vxlan)
      nssm install kube-proxy C:\k\kube-proxy.exe
      nssm set kube-proxy AppDirectory c:\k
      nssm set kube-proxy AppParameters --v=4 --proxy-mode=kernelspace --feature-gates="WinOverlay=true" --hostname-override=<hostname> --kubeconfig=c:\k\config --network-name=vxlan0 --source-vip=<source-vip> --enable-dsr=false --log-dir=<log directory> --logtostderr=false
      nssm set kube-proxy DependOnService kubelet
      nssm start kube-proxy

      For initial troubleshooting, you can use the following flags in nssm.exe to redirect stdout and stderr to a output file:

      nssm set <Service Name> AppStdout C:\k\mysvc.log
      nssm set <Service Name> AppStderr C:\k\mysvc.log

      For additional details, see official nssm usage docs.

  3. My Windows Pods do not have network connectivity

    If you are using virtual machines, ensure that MAC spoofing is enabled on all the VM network adapter(s).

  4. My Windows Pods cannot ping external resources

    Windows Pods do not have outbound rules programmed for the ICMP protocol today. However, TCP/UDP is supported. When trying to demonstrate connectivity to resources outside of the cluster, please substitute ping <IP> with corresponding curl <IP> commands.

    If you are still facing problems, most likely your network configuration in cni.conf deserves some extra attention. You can always edit this static file, the configuration will be applied to any newly created Kubernetes resources.

    One of the Kubernetes networking requirements (see Kubernetes model) is for cluster communication to occur without NAT internally. To honor this requirement, there is an ExceptionList for all the communication where we do not want outbound NAT to occur. However, this also means that you need to exclude the external IP you are trying to query from the ExceptionList. Only then will the traffic originating from your Windows pods be SNAT’ed correctly to receive a response from the outside world. In this regard, your ExceptionList in cni.conf should look as follows:

    "ExceptionList": [
                    "10.244.0.0/16",  # Cluster subnet
                    "10.96.0.0/12",   # Service subnet
                    "10.127.130.0/24" # Management (host) subnet
                ]
  5. My Windows node cannot access NodePort service

    Local NodePort access from the node itself will fail. This is a known limitation. NodePort access will work from other nodes or external clients.

  6. vNICs and HNS endpoints of containers are being deleted

    This issue can be caused when the hostname-override parameter is not passed to kube-proxy. To resolve it, users need to pass the hostname to kube-proxy as follows:

    C:\k\kube-proxy.exe --hostname-override=$(hostname)
  7. With flannel my nodes are having issues after rejoining a cluster

    Whenever a previously deleted node is being re-joined to the cluster, flannelD will try to assign a new pod subnet to the node. Users should remove the old pod subnet configuration files in the following paths:

    Remove-Item C:\k\SourceVip.json
    Remove-Item C:\k\SourceVipRequest.json
  8. After launching start.ps1, flanneld is stuck in “Waiting for the Network to be created”

    There are numerous reports of this issue which are being investigated; most likely it is a timing issue for when the management IP of the flannel network is set. A workaround is to simply relaunch start.ps1 or relaunch it manually as follows:

    PS C:> [Environment]::SetEnvironmentVariable("NODE_NAME", "<Windows_Worker_Hostname>")
    PS C:> C:\flannel\flanneld.exe --kubeconfig-file=c:\k\config --iface=<Windows_Worker_Node_IP> --ip-masq=1 --kube-subnet-mgr=1
  9. My Windows Pods cannot launch because of missing /run/flannel/subnet.env

    This indicates that Flannel didn’t launch correctly. You can either try to restart flanneld.exe or you can copy the files over manually from /run/flannel/subnet.env on the Kubernetes master toC:\run\flannel\subnet.env on the Windows worker node and modify the FLANNEL_SUBNET row to a different number. For example, if node subnet 10.244.4.1/24 is desired:

    FLANNEL_NETWORK=10.244.0.0/16
    FLANNEL_SUBNET=10.244.4.1/24
    FLANNEL_MTU=1500
    FLANNEL_IPMASQ=true
  10. My Windows node cannot access my services using the service IP

    This is a known limitation of the current networking stack on Windows. Windows Pods are able to access the service IP however.

  11. No network adapter is found when starting kubelet

    The Windows networking stack needs a virtual adapter for Kubernetes networking to work. If the following commands return no results (in an admin shell), virtual network creation — a necessary prerequisite for Kubelet to work — has failed:

    Get-HnsNetwork | ? Name -ieq "cbr0"
    Get-NetAdapter | ? Name -Like "vEthernet (Ethernet*"

    Often it is worthwhile to modify the InterfaceName parameter of the start.ps1 script, in cases where the host’s network adapter isn’t “Ethernet”. Otherwise, consult the output of the start-kubelet.ps1 script to see if there are errors during virtual network creation.

  12. My Pods are stuck at “Container Creating” or restarting over and over

    Check that your pause image is compatible with your OS version. The instructions assume that both the OS and the containers are version 1803. If you have a later version of Windows, such as an Insider build, you will need to adjust the images accordingly. Please refer to the Microsoft’s Docker repository for images. Regardless, both the pause image Dockerfile and the sample service will expect the image to be tagged as :latest.

Further investigation

Check the DNS limitations for Windows in this section.

If these steps don’t resolve your problem, you can get help running Windows containers on Windows nodes in Kubernetes through:

Reporting Issues and Feature Requests

If you have what looks like a bug, or you would like to make a feature request, please use the Github issue tracking system. You can open issues on GitHub and assign them to SIG-Windows. You should first search the list of issues in case it was reported previously and comment with your experience on the issue and add additional logs. SIG-Windows Slack is also a great avenue to get some initial support and troubleshooting ideas prior to creating a ticket.

If filing a bug, please include detailed information about how to reproduce the problem, such as:

What's next

We have a lot of features in our roadmap. An abbreviated high level list is included below, but we encourage you to view our roadmap project and help us make Windows support better by contributing.

CRI-ContainerD

ContainerD is another OCI-compliant runtime that recently graduated as a CNCF project. It’s currently tested on Linux, but 1.3 will bring support for Windows and Hyper-V. [reference]

The CRI-ContainerD interface will be able to manage sandboxes based on Hyper-V. This provides a foundation where RuntimeClasses could be implemented for new use cases including:

Deployment with kubeadm and cluster API

Kubeadm is becoming the de facto standard for users to deploy a Kubernetes cluster. Windows node support in kubeadm will come in a future release. We are also making investments in cluster API to ensure Windows nodes are properly provisioned.

A few other key features

Feedback