User Data Configuration

This document describes how you can customize the UserData that will be specified on your EC2 worker nodes, without using a launch template.

Configuration

In order to specify custom user data, you must include it within a AWSNodeTemplate resource. You can then reference this AWSNodeTemplate resource through spec.providerRef in your provisioner.

Examples

Your UserData can be added to spec.userData in the AWSNodeTemplate resource like this -

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: bottlerocket-example
spec:
  amiFamily: Bottlerocket
  instanceProfile: MyInstanceProfile
  subnetSelector:
    karpenter.sh/discovery: my-cluster
  securityGroupSelector:
    karpenter.sh/discovery: my-cluster
  userData:  |
    [settings.kubernetes]
    kube-api-qps = 30
    [settings.kubernetes.eviction-hard]
    "memory.available" = "20%"    

For more examples on configuring UserData, see the examples for AL2 and Bottlerocket.

UserData Content and Merge Semantics

Karpenter will evaluate and merge the UserData that you specify in the AWSNodeTemplate resources depending upon the AMIFamily that you have chosen.

Bottlerocket

• Your UserData must be valid TOML.
• Karpenter will automatically merge settings to ensure successful bootstrap including cluster-name, api-server and cluster-certificate. Any labels and taints that need to be set based on pod requirements will also be specified in the final merged UserData.
  • All Kubelet settings that Karpenter applies will override the corresponding settings in the provided UserData. For example, if you’ve specified settings.kubernetes.cluster-name, it will be overridden.
  • If MaxPods is specified via the binary arg to Karpenter, the value will override anything specified in the UserData.
  • If ClusterDNS is specified via spec.kubeletConfiguration, then that value will override anything specified in the UserData.
• Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter.

Consider the following example to understand how your custom UserData settings will be merged in.

Your UserData -

[settings.kubernetes.eviction-hard]
"memory.available" = "12%"
[settings.kubernetes]
"unknown-setting" = "unknown"
[settings.kubernetes.node-labels]
'field.controlled.by/karpenter': 'will-be-overridden'

Final merged UserData -

[settings]
[settings.kubernetes]
api-server = 'https://cluster'
cluster-certificate = 'ca-bundle'
cluster-name = 'cluster'

[settings.kubernetes.node-labels]
'karpenter.sh/capacity-type' = 'on-demand'
'karpenter.sh/provisioner-name' = 'provisioner'

[settings.kubernetes.node-taints]

[settings.kubernetes.eviction-hard]
'memory.available' = '12%%'

AL2 and Ubuntu

• Your UserData must be in the MIME multi part archive format.
• Karpenter will merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script.
  • Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before.

Consider the following example to understand how your custom UserData will be merged -

Your UserData -

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script"

--BOUNDARY--

The final merged UserData that will be applied to your worker nodes -

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script"

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--container-runtime containerd \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/provisioner-name=test  --max-pods=110'
--//--