开发K8s准入控制器前的准备工作：集群检查与项目搭建指南

前言在上一篇文章中我们规划了要开发一个自动注入Nginx Sidecar的Webhook。但在真正开始写代码之前必须先做好充分的准备工作。我曾经踩过一个坑代码写完了部署到集群却发现apiserver根本没有启用MutatingAdmissionWebhook插件导致Webhook完全不生效。回过来检查配置、重新编译apiserver浪费了大量时间。今天就详细讲讲开发K8s准入控制器前的准备工作包括集群环境检查、项目初始化、配置设计等。第一步检查集群准入配置1.1 检查Admission Registration API首先确认集群是否启用了准入控制的APIkubectl api-versions|grepadmission期望的输出admissionregistration.k8s.io/v1 admissionregistration.k8s.io/v1beta1说明v1是稳定版本Kubernetes 1.16v1beta1是旧版本建议用v1如果没有输出说明集群版本太旧1.9不支持Webhook1.2 检查准入控制插件确认apiserver启用了MutatingAdmissionWebhook和ValidatingAdmissionWebhook# 方法1查看apiserver进程参数psaux|grepkube-apiserver|grepenable-admission-plugins# 方法2如果是静态Podkubectl get pod-nkube-system-lcomponentkube-apiserver-oyaml|grepenable-admission-plugins# 方法3查看kube-apiserver帮助如果在本地有apiserver二进制文件kube-apiserver-h|grepenable-admission-pluginsKubernetes 1.20默认启用的插件NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota关键插件✅MutatingAdmissionWebhook必须启用用于Mutating Webhook✅ValidatingAdmissionWebhook必须启用用于Validating Webhook如果发现没有启用# 修改apiserver启动参数kube-apiserver\--enable-admission-plugins...,MutatingAdmissionWebhook,ValidatingAdmissionWebhook# 如果是kubeadm部署的集群修改/etc/kubernetes/manifests/kube-apiserver.yaml1.3 测试Webhook连通性可以先创建一个简单的测试Webhook验证环境# 创建一个简单的nginx服务模拟Webhookkubectl create deployment webhook-test--imagenginx kubectl expose deployment webhook-test--port443--target-port443# 检查是否可以访问kubectl get svc webhook-test第二步创建Go项目2.1 初始化项目# 创建项目目录mkdir-p~/projects/kube-mutating-webhook-inject-podcd~/projects/kube-mutating-webhook-inject-pod# 初始化Go模块go mod init kube-mutating-webhook-inject-pod# 创建项目结构mkdir-ppkg deploy certs2.2 添加依赖# 添加K8s API依赖go get k8s.io/apilatest go get k8s.io/apimachinerylatest# 添加YAML解析依赖go get gopkg.in/yaml.v2# 添加日志依赖可选也可以用标准库go get github.com/golang/glog# 整理依赖go mod tidy生成的go.mod文件module kube-mutating-webhook-inject-podgo1.21require(github.com/golang/glog v1.2.0gopkg.in/yaml.v2 v2.4.0k8s.io/api v0.28.0k8s.io/apimachinery v0.28.0)第三步设计配置结构3.1 配置文件设计我们需要一个配置文件定义要注入的Sidecar容器。复用K8s原生的Container和Volume结构# config.yamlcontainers:-name:sidecar-nginximage:nginx:1.25-alpineimagePullPolicy:IfNotPresentports:-containerPort:80protocol:TCPvolumeMounts:-name:nginx-confmountPath:/etc/nginx/conf.dresources:limits:cpu:100mmemory:128Mirequests:cpu:50mmemory:64Mivolumes:-name:nginx-confconfigMap:name:nginx-sidecar-config设计说明containers要注入的sidecar容器列表可以注入多个volumessidecar需要的volume使用K8s原生结构便于验证和使用3.2 Go结构体定义// pkg/config.gopackagemainimport(corev1k8s.io/api/core/v1)// Config 存储注入配置typeConfigstruct{Containers[]corev1.Containeryaml:containers// 要注入的容器Volumes[]corev1.Volumeyaml:volumes// 要注入的volume}3.3 配置加载函数// pkg/config.goimport(crypto/sha256fmtio/ioutilgithub.com/golang/gloggopkg.in/yaml.v2)// loadConfig 从文件加载配置funcloadConfig(configFilestring)(*Config,error){// 读取配置文件data,err:ioutil.ReadFile(configFile)iferr!nil{returnnil,fmt.Errorf(failed to read config file: %w,err)}// 计算配置文件的hash便于调试glog.Infof(New configuration: sha256sum %x,sha256.Sum256(data))// 解析YAMLvarcfg Configiferr:yaml.Unmarshal(data,cfg);err!nil{returnnil,fmt.Errorf(failed to parse config: %w,err)}returncfg,nil}第四步搭建HTTPS服务器Webhook必须使用HTTPSKubernetes要求所以我们需要TLS证书后续会生成HTTPS服务器4.1 Webhook配置结构// pkg/webhook.gopackagemainimport(net/http)// webhookServer Webhook服务器typewebhookServerstruct{sidecarConfig*Config// 注入配置server*http.Server// HTTP服务器}4.2 命令行参数// main.gopackagemainimport(flagfmtgithub.com/golang/glog)// webHookSvrOptions Webhook服务器选项typewebHookSvrOptionsstruct{portint// HTTPS监听端口certFilestring// TLS证书文件路径keyFilestring// TLS私钥文件路径sidecarCfgFilestring// Sidecar配置文件路径}funcmain(){varrunOption webHookSvrOptions// 解析命令行参数flag.IntVar(runOption.port,port,8443,Webhook server port.)flag.StringVar(runOption.certFile,tlsCertFile,/etc/webhook/certs/cert.pem,File containing the x509 Certificate for HTTPS.)flag.StringVar(runOption.keyFile,tlsKeyFile,/etc/webhook/certs/key.pem,File containing the x509 private key to --tlsCertFile.)flag.StringVar(runOption.sidecarCfgFile,sidecarCfgFile,config.yaml,File containing the mutation configuration.)flag.Parse()// 加载配置sidecarConfig,err:loadConfig(runOption.sidecarCfgFile)iferr!nil{glog.Errorf(Failed to load configuration: %v,err)return}glog.Infof([sidecarConfig:%v],sidecarConfig)}4.3 加载TLS证书// main.goimport(crypto/tls)funcmain(){// ... 加载配置 ...// 加载TLS证书对pair,err:tls.LoadX509KeyPair(runOption.certFile,runOption.keyFile)iferr!nil{glog.Errorf(Failed to load key pair: %v,err)return}glog.Infof(Loaded TLS certificate successfully)}4.4 创建HTTPS服务器// main.goimport(fmtnet/http)funcmain(){// ... 加载配置和证书 ...// 创建Webhook服务器实例webhooksvr:webhookServer{sidecarConfig:sidecarConfig,server:http.Server{Addr:fmt.Sprintf(:%v,runOption.port),TLSConfig:tls.Config{Certificates:[]tls.Certificate{pair},},},}// 创建路由mux:http.NewServeMux()mux.HandleFunc(/mutate,webhooksvr.serveMutate)mux.HandleFunc(/health,webhooksvr.serveHealth)webhooksvr.server.Handlermux glog.Infof(Starting webhook server on port %d,runOption.port)// 在goroutine中启动服务器gofunc(){// 注意ListenAndServeTLS的第一个参数为空字符串使用server.Addriferr:webhooksvr.server.ListenAndServeTLS(,);err!nil{glog.Errorf(Failed to listen and serve webhook server: %v,err)}}()}4.5 添加Handler// pkg/webhook.goimport(ionet/httpgithub.com/golang/glog)// serveMutate 处理/mutate请求func(ws*webhookServer)serveMutate(w http.ResponseWriter,r*http.Request){// 读取请求体body,err:io.ReadAll(r.Body)iferr!nil{http.Error(w,fmt.Sprintf(could not read request body: %v,err),http.StatusBadRequest)return}glog.Infof(Received mutation request: %s,string(body))// TODO: 解析AdmissionReview构造响应// 这部分在下一篇文章中详细讲解}// serveHealth 健康检查func(ws*webhookServer)serveHealth(w http.ResponseWriter,r*http.Request){w.WriteHeader(http.StatusOK)w.Write([]byte(ok))}4.6 优雅关闭// main.goimport(contextosos/signalsyscall)funcmain(){// ... 启动服务器 ...// 监听系统信号实现优雅关闭signalChan:make(chanos.Signal,1)signal.Notify(signalChan,syscall.SIGINT,syscall.SIGTERM)// 等待信号-signalChan glog.Infof(Got OS shutdown signal, shutting down webhook server gracefully...)// 优雅关闭服务器ctx,cancel:context.WithTimeout(context.Background(),10*time.Second)defercancel()iferr:webhooksvr.server.Shutdown(ctx);err!nil{glog.Errorf(Server shutdown error: %v,err)}}第五步本地测试准备5.1 创建测试证书临时在正式部署前可以用自签名证书本地测试# 生成私钥openssl genrsa-outcerts/server.key2048# 生成证书签名请求openssl req-new-keycerts/server.key-outcerts/server.csr\-subj/CNlocalhost/OTest# 生成自签名证书openssl x509-req-days365-incerts/server.csr\-signkeycerts/server.key-outcerts/server.crt5.2 创建测试配置# test-config.yamlcontainers:-name:sidecar-nginximage:nginx:alpineports:-containerPort:80volumes:[]5.3 本地运行测试# 编译go build-owebhook-server.# 运行使用测试证书./webhook-server\--port8443\--tlsCertFilecerts/server.crt\--tlsKeyFilecerts/server.key\--sidecarCfgFiletest-config.yaml# 测试健康检查curl-khttps://localhost:8443/health# 输出: ok第六步容器化准备6.1 Dockerfile# Dockerfile FROM golang:1.21-alpine AS builder WORKDIR /app COPY go.mod go.sum ./ RUN go mod download COPY . . RUN CGO_ENABLED0 GOOSlinux go build -o webhook-server . FROM alpine:latest RUN apk --no-cache add ca-certificates WORKDIR / COPY --frombuilder /app/webhook-server /webhook-server # 创建证书和配置目录 RUN mkdir -p /etc/webhook/certs /etc/webhook/config EXPOSE 8443 ENTRYPOINT [/webhook-server]6.2 构建镜像# 构建dockerbuild-tnginx-sidecar-injector:v1.0.# 测试运行dockerrun-d\-p8443:8443\-v$(pwd)/certs:/etc/webhook/certs\-v$(pwd)/test-config.yaml:/etc/webhook/config/config.yaml\nginx-sidecar-injector:v1.0\--sidecarCfgFile/etc/webhook/config/config.yaml准备工作清单在开始写核心业务逻辑前确保完成了以下检查集群启用了admissionregistration.k8s.io/v1API集群启用了MutatingAdmissionWebhook和ValidatingAdmissionWebhook插件创建了Go项目并初始化了模块设计了配置文件结构和加载逻辑搭建了基本的HTTPS服务器框架实现了配置加载功能实现了健康检查接口创建了Dockerfile用于容器化本地测试可以正常运行常见问题排查问题1kubectl api-versions没有admissionregistration原因集群版本太旧1.9解决升级Kubernetes版本问题2加载证书失败Failed to load key pair: open /etc/webhook/certs/cert.pem: no such file or directory原因证书路径错误或证书不存在解决# 检查证书文件是否存在ls-lacerts/# 确保路径正确./webhook-server--tlsCertFile./certs/server.crt--tlsKeyFile./certs/server.key问题3端口被占用Failed to listen and serve webhook server: listen :8443: bind: address already in use解决# 查找占用端口的进程lsof-i:8443# 杀掉进程或更换端口./webhook-server--port8444问题4配置文件解析失败Failed to load configuration: failed to parse config: yaml: unmarshal errors原因YAML格式错误解决使用YAML验证工具检查配置文件总结完成准备工作后项目结构应该是kube-mutating-webhook-inject-pod/ ├── go.mod ├── go.sum ├── main.go # 主程序入口 ├── config.yaml # 配置文件 ├── pkg/ │ └── webhook.go # Webhook逻辑待实现 ├── certs/ # 证书目录 │ ├── server.crt │ └── server.key ├── deploy/ # 部署文件待创建 └── Dockerfile现在基础框架已经搭建好了下一篇文章我们将实现核心的Mutation逻辑——解析AdmissionReview、构造JSON Patch、返回响应。下一步准备好环境后就可以开始实现核心的Webhook逻辑了解析AdmissionReview请求构造Pod Patch返回AdmissionResponse部署到K8s集群你准备好了吗你的集群启用了MutatingAdmissionWebhook吗你是如何管理项目依赖的Go Modules还是其他工具你在搭建HTTPS服务器时遇到过什么问题求助与交流如果你在准备工作中遇到了问题欢迎在评论区交流。