controller-tools实战：自动生成代码

初始化项目

初始化go项目，并get client-go

cd controller-tools-demo
go mod init controller-tools-demo

go get k8s.io/client-go
go get k8s.io/apimachinery

使用type-scaffold工具生成types.go

需要注意：
- type-scaffold并不会生成文件，而是生成types.go的内容，打印到控制台，我们需要手动copy到types.go文件中去
- 不过使用kubebuilder的时候，会帮我们生成types.go文件的

执行 type-scaffold –kind=Application，得到types.go的内容

[root@master controller-tools-demo]# type-scaffold --kind=Application
// ApplicationSpec defines the desired state of Application
type ApplicationSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
}

// ApplicationStatus defines the observed state of Application.
// It should always be reconstructable from the state of the cluster and/or outside world.
type ApplicationStatus struct {
        // INSERT ADDITIONAL STATUS FIELDS -- observed state of cluster
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// Application is the Schema for the applications API
// +k8s:openapi-gen=true
type Application struct {
        metav1.TypeMeta   `json:",inline"`
        metav1.ObjectMeta `json:"metadata,omitempty"`

        Spec   ApplicationSpec   `json:"spec,omitempty"`
        Status ApplicationStatus `json:"status,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// ApplicationList contains a list of Application
type ApplicationList struct {
        metav1.TypeMeta `json:",inline"`
        metav1.ListMeta `json:"metadata,omitempty"`
        Items           []Application `json:"items"`
}

在 v1alpha1 目录下创建 types.go 文件，将控制台的内容copy进去，记得导包

package v1alpha1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

// ApplicationSpec defines the desired state of Application
type ApplicationSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
}

// ApplicationStatus defines the observed state of Application.
// It should always be reconstructable from the state of the cluster and/or outside world.
type ApplicationStatus struct {
	// INSERT ADDITIONAL STATUS FIELDS -- observed state of cluster
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// Application is the Schema for the applications API
// +k8s:openapi-gen=true
type Application struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   ApplicationSpec   `json:"spec,omitempty"`
	Status ApplicationStatus `json:"status,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// ApplicationList contains a list of Application
type ApplicationList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Application `json:"items"`
}

使用controller-gen生成deepcopy和crd文件

controller-gen –help 查看帮助文档

帮助文档给出了很多 examples

[root@master v1alpha1]# controller-gen --help
Generate Kubernetes API extension resources and code.

Usage:
  controller-gen [flags]

Examples:
        # Generate RBAC manifests and crds for all types under apis/,
        # outputting crds to /tmp/crds and everything else to stdout
        controller-gen rbac:roleName=<role name> crd paths=./apis/... output:crd:dir=/tmp/crds output:stdout

        # Generate deepcopy/runtime.Object implementations for a particular file
        controller-gen object paths=./apis/v1beta1/some_types.go

        # Generate OpenAPI v3 schemas for API packages and merge them into existing CRD manifests
        controller-gen schemapatch:manifests=./manifests output:dir=./manifests paths=./pkg/apis/...

        # Run all the generators for a given project
        controller-gen paths=./apis/...

        # Explain the markers for generating CRDs, and their arguments
        controller-gen crd -ww


Flags:
  -h, --detailed-help count   print out more detailed help
                              (up to -hhh for the most detailed output, or -hhhh for json output)
      --help                  print out usage and a summary of options
      --version               show version
  -w, --which-markers count   print out all markers available with the requested generators
                              (up to -www for the most detailed output, or -wwww for json output)


Options


generators

+webhook                                                                                                                                           package  generates (partial) {Mutating,Validating}WebhookConfiguration objects.
+schemapatch[:generateEmbeddedObjectMeta=<bool>],manifests=<string>[,maxDescLen=<int>]                                                             package  patches existing CRDs with new schemata.
+rbac:roleName=<string>                                                                                                                            package  generates ClusterRole objects.
+object[:headerFile=<string>][,year=<string>]                                                                                                      package  generates code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
+crd[:allowDangerousTypes=<bool>][,crdVersions=<[]string>][,generateEmbeddedObjectMeta=<bool>][,ignoreUnexportedFields=<bool>][,maxDescLen=<int>]  package  generates CustomResourceDefinition objects.


generic

+paths=<[]string>  package  represents paths and go-style path patterns to use as package roots.


output rules (optionally as output:<generator>:...)

+output:artifacts[:code=<string>],config=<string>  package  outputs artifacts to different locations, depending on whether they're package-associated or not.
+output:dir=<string>                               package  outputs each artifact to the given directory, regardless of if it's package-associated or not.
+output:none                                       package  skips outputting anything.                                             
+output:stdout                                     package  outputs everything to standard-out, with no separation.

使用controller-gen生成deepcopy

先刷新一下包
1
go mod tidy

执行命令

1 2	cd controller-tools-demo controller-gen object paths=pkg/apis/appcontroller/v1alpha1/types.go

执行后，查看目录文件，发现生成了一个 zz_generated.deepcopy.go 文件

[root@master controller-tools-demo]# tree
.
├── go.mod
├── go.sum
└── pkg
    └── apis
        └── appcontroller
            └── v1alpha1
                ├── types.go
                └── zz_generated.deepcopy.go

查看 zz_generated.deepcopy.go 文件内容

//go:build !ignore_autogenerated
// +build !ignore_autogenerated

// Code generated by controller-gen. DO NOT EDIT.

package v1alpha1

import (
	runtime "k8s.io/apimachinery/pkg/runtime"
)

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Application) DeepCopyInto(out *Application) {
	*out = *in
	out.TypeMeta = in.TypeMeta
	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
	out.Spec = in.Spec
	out.Status = in.Status
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Application.
func (in *Application) DeepCopy() *Application {
	if in == nil {
		return nil
	}
	out := new(Application)
	in.DeepCopyInto(out)
	return out
}

// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *Application) DeepCopyObject() runtime.Object {
	if c := in.DeepCopy(); c != nil {
		return c
	}
	return nil
}

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *ApplicationList) DeepCopyInto(out *ApplicationList) {
	*out = *in
	out.TypeMeta = in.TypeMeta
	in.ListMeta.DeepCopyInto(&out.ListMeta)
	if in.Items != nil {
		in, out := &in.Items, &out.Items
		*out = make([]Application, len(*in))
		for i := range *in {
			(*in)[i].DeepCopyInto(&(*out)[i])
		}
	}
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApplicationList.
func (in *ApplicationList) DeepCopy() *ApplicationList {
	if in == nil {
		return nil
	}
	out := new(ApplicationList)
	in.DeepCopyInto(out)
	return out
}

// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *ApplicationList) DeepCopyObject() runtime.Object {
	if c := in.DeepCopy(); c != nil {
		return c
	}
	return nil
}

使用controller-gen生成crd

先刷新一下包
1
go mod tidy

执行命令

1 2	cd controller-tools-demo controller-gen crd paths=./... output:crd:dir=config/crd

paths=./… 表示将当前目录下的所有子目录都包括在生成过程中
output:crd:dir=config/crd 指定了输出目录为 config/crd

执行后，生成目录 config 和文件 _.yaml

[root@master controller-tools-demo]# tree
.
├── config
│   └── crd
│       └── _.yaml
├── go.mod
├── go.sum
└── pkg
    └── apis
        └── appcontroller
            └── v1alpha1
                ├── types.go
                └── zz_generated.deepcopy.go

文件 _.yaml 没有名字？指定groupName后重新生成 crd 文件
- 因为我们没有给它指定 groupName，所以生成的yaml文件默认没有名字
- 我们在 pkg/apis/appcontroller/v1alpha1 下创建一个 doc.go 文件，并在里面写上如下内容
  1
  2
  // +groupName=appcontroller.k8s.io
  package v1alpha1

然后删除原config后，重新生成crd

1
2
3

cd controller-tools-demo
rm -rf config
controller-gen crd paths=./... output:crd:dir=config/crd

生成完成后，目录如下

crd文件名称为：appcontroller.k8s.io_applications.yaml

[root@master controller-tools-demo]# tree
.
├── config
│   └── crd
│       └── appcontroller.k8s.io_applications.yaml
├── go.mod
├── go.sum
└── pkg
    └── apis
        └── appcontroller
            └── v1alpha1
                ├── doc.go
                ├── types.go
                └── zz_generated.deepcopy.go

6 directories, 6 files

appcontroller.k8s.io_applications.yaml 内容如下

可以看到，openAPIV3Schema.properties.spec 下面，没有 properties 项，因为我们的ApplicationSpec 为空，内部一个属性都没有。

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: (devel)
  creationTimestamp: null
  name: applications.appcontroller.k8s.io
spec:
  group: appcontroller.k8s.io
  names:
    kind: Application
    listKind: ApplicationList
    plural: applications
    singular: application
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Application is the Schema for the applications API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: ApplicationSpec defines the desired state of Application
            type: object
          status:
            description: ApplicationStatus defines the observed state of Application.
              It should always be reconstructable from the state of the cluster and/or
              outside world.
            type: object
        type: object
    served: true
    storage: true

为ApplicationSpec添加内容，并重新生成crd文件

修改types.go，在 ApplicationSpec 中添加两个字段

type ApplicationSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
	Name     string `json:"name"`
	Replicas int32  `json:"replicas"`
}

删除原config后，重新生成crd文件

1
2
3

cd controller-tools-demo
rm -rf config
controller-gen crd paths=./... output:crd:dir=config/crd

新的crd文件内容如下：

可以看到，openAPIV3Schema.properties.spec 下面，出现了 properties 项，因为我们为ApplicationSpec 添加了Name、Replicas两个值。

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: (devel)
  creationTimestamp: null
  name: applications.appcontroller.k8s.io
spec:
  group: appcontroller.k8s.io
  names:
    kind: Application
    listKind: ApplicationList
    plural: applications
    singular: application
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Application is the Schema for the applications API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: ApplicationSpec defines the desired state of Application
            properties:
              name:
                description: INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
                type: string
              replicas:
                format: int32
                type: integer
            required:
            - name
            - replicas
            type: object
          status:
            description: ApplicationStatus defines the observed state of Application.
              It should always be reconstructable from the state of the cluster and/or
              outside world.
            type: object
        type: object
    served: true
    storage: true

手动注册版本v1alpha1的CRD资源

在生成了客户端代码后，我们还需要手动注册版本v1alpha1的CRD资源，才能真正使用这个client，不然在编译时会出现 undefined: v1alpha1.AddToScheme 错误、undefined: v1alpha1.Resource 错误。
v1alpha1.AddToScheme、v1alpha1.Resource 这两个是用于 client 注册的

编写 v1alpha1/register.go

package v1alpha1

import (
	"controller-tools-demo/pkg/apis/appcontroller"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{Group: appcontroller.GroupName, Version: "v1alpha1"}

// Kind takes an unqualified kind and returns back a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
	return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

var (
	// SchemeBuilder initializes a scheme builder
	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
	// AddToScheme is a global function that registers this API group & version to a scheme
	AddToScheme = SchemeBuilder.AddToScheme
)

// Adds the list of known types to Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&Application{},
		&ApplicationList{},
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}

在kubernetes集群中应用CRD

kubectl apply crd 资源

上面我们已经使用 controller-gen 自动生成了 CRD 文件，名称为 appcontroller.k8s.io_applications.yaml，再 config/crd 目录下

我们需要使用 kubectl apply 命令创建CR资源

1 2	cd controller-tools-demo kubectl apply -f config/crd/appcontroller.k8s.io_applications.yaml

有可能报错：

The CustomResourceDefinition "applications.appcontroller.k8s.io" is invalid: metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111

解决方法
- 这是kubernetes的保护机制，防止外部随意创建crd，破坏环境
- 报错中已经给了提示，查看github：https://github.com/kubernetes/enhancements/pull/1111
- 只需要在crd中，添加一条 annotation，然后再执行 kubectl apply -f 命令就可以了
  1
  2
  3
  metadata:
  annotations:
  api-approved.kubernetes.io: "https://github.com/kubernetes/kubernetes/pull/78458"

添加之后，完整的 crd 文件内容如下：

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: (devel)
    api-approved.kubernetes.io: "https://github.com/kubernetes/kubernetes/pull/78458"
  creationTimestamp: null
  name: applications.appcontroller.k8s.io
spec:
  group: appcontroller.k8s.io
  names:
    kind: Application
    listKind: ApplicationList
    plural: applications
    singular: application
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Application is the Schema for the applications API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: ApplicationSpec defines the desired state of Application
            properties:
              name:
                description: INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
                type: string
              replicas:
                format: int32
                type: integer
            required:
            - name
            - replicas
            type: object
          status:
            description: ApplicationStatus defines the observed state of Application.
              It should always be reconstructable from the state of the cluster and/or
              outside world.
            type: object
        type: object
    served: true
    storage: true

编写 crd 资源的 example

在 config 目录下，创建一个example目录，内部可以编写一些 application资源的yaml，用于测试Application资源的创建是否可以成功

在config/example目录下，编写一个test_app.yaml

apiVersion: appcontroller.k8s.io/v1alpha1
kind: Application
metadata:
  name: testapp
  namespace: tcs
spec:
  name: testapp1
  replicas: 2

创建Application资源

1 2	cd controller-tools-demo kubectl apply -f ./config/example/test_app.yaml

测试Application资源的获取

在项目根目录下，创建一个cmd目录，里面创建一个main.go文件
下面我们演示如何使用 code-generator 为 Application 的 v1alpha1 生成的客户端 AppcontrollerV1alpha1Client

编写main.go

package main

import (
	"context"
	"controller-tools-demo/pkg/apis/appcontroller/v1alpha1"
	"fmt"
	"log"

	"k8s.io/client-go/kubernetes/scheme"

	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/clientcmd"
)

func main() {
	config, err := clientcmd.BuildConfigFromFlags("", clientcmd.RecommendedHomeFile)
	if err != nil {
		log.Fatalln(err)
	}

	config.APIPath = "/apis/"
	config.GroupVersion = &v1alpha1.SchemeGroupVersion
	config.NegotiatedSerializer = scheme.Codecs

	client, err := rest.RESTClientFor(config)
	if err != nil {
		log.Fatalln(err)
	}

	app := v1alpha1.Application{}
	err = client.Get().Namespace("tcs").Resource("applications").Name("testapp").Do(context.TODO()).Into(&app)
	if err != nil {
		log.Fatalln(err)
	}

	newObj := app.DeepCopy()
	newObj.Spec.Name = "testapp2"

	fmt.Println(app.Spec.Name)
	fmt.Println(app.Spec.Replicas)

	fmt.Println(newObj.Spec.Name)
}