需求分析

  • 设计一个CRD,名称为 App:在AppSpec中,可以包含一个 Deployment 的 部分属性、一个 Service 的 部分属性
  • 为App编写一个控制器 AppController:当创建App的时候,控制器AppController会检查AppSpec,如果包含 Deployment 或 Service,就创建对应的 Deployment 或 Service

初始化项目

初始化项目

初始化go项目,并get client-go

1
2
3
4
5
cd crd-controller-demo
go mod init crd-controller-demo

go get k8s.io/client-go
go get k8s.io/apimachinery

编写boilerplate.go.txt

该文件是文件开头统一的注释,会在使用 code-generator 脚本时,指定 boilerplate.go.txt 文件的所在目录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
/*
Copyright 2019 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

@Time : 2024/2
@Author : grahamzhu
@Software: GoLand
*/

编写tools.go

我们要使用 code-generator,可代码中还没有任何位置 导入过 code-generator 的包,所以我们需要一个类,专门用于将 code-generator 的包导入。一般使用tools.go来做这件事

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
//go:build tools
// +build tools

/*
Copyright 2019 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

// This package imports things required by build scripts, to force `go mod` to see them as dependencies
package tools

import _ "k8s.io/code-generator"

编写appcontroller/v1/register.go

注意,这是appcontroller目录下的register.go,并非某个版本目录下的,版本目录下的,再使用 code-generator 脚本后,再进行编写

1
2
3
4
5
package appcontroller

const (
GroupName = "appcontroller.k8s.io"
)

编写doc.go

1
2
3
4
5
// +k8s:deepcopy-gen=package
// +groupName=appcontroller.k8s.io

// Package v1 v1版本的api包
package v1

自动生成代码

使用type-scaffold工具生成 types.go

1、生成types.go

  • 需要注意:
    • type-scaffold并不会生成文件,而是生成types.go的内容,打印到控制台,我们需要手动copy到types.go文件中去
    • 不过使用kubebuilder的时候,会帮我们生成types.go文件的
  • 执行 type-scaffold –kind=App,得到types.go的内容
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    [root@master controller-tools-demo]# type-scaffold --kind=App
    // AppSpec defines the desired state of App
    type AppSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
    }

    // AppStatus defines the observed state of App.
    // It should always be reconstructable from the state of the cluster and/or outside world.
    type AppStatus struct {
    // INSERT ADDITIONAL STATUS FIELDS -- observed state of cluster
    }

    // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

    // App is the Schema for the apps API
    // +k8s:openapi-gen=true
    type App struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec AppSpec `json:"spec,omitempty"`
    Status AppStatus `json:"status,omitempty"`
    }

    // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

    // AppList contains a list of App
    type AppList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items []App `json:"items"`
    }
    在 v1 目录下创建 types.go 文件,将控制台的内容copy进去,记得导包
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    package v1

    import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

    // AppSpec defines the desired state of App
    type AppSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS -- desired state of cluster
    }

    // AppStatus defines the observed state of App.
    // It should always be reconstructable from the state of the cluster and/or outside world.
    type AppStatus struct {
    // INSERT ADDITIONAL STATUS FIELDS -- observed state of cluster
    }

    // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

    // App is the Schema for the apps API
    // +k8s:openapi-gen=true
    type App struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec AppSpec `json:"spec,omitempty"`
    Status AppStatus `json:"status,omitempty"`
    }

    // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

    // AppList contains a list of App
    type AppList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items []App `json:"items"`
    }

2、修改types.go文件,并添加自动生成标签

修改types.go,并在types.go的struct上面,需要加上一些·client的标签,才能使用code-generator自动生成client
修改后,typs.go文件完整内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
package v1

import (
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// App is the Schema for the apps API
type App struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`

Spec AppSpec `json:"spec,omitempty"`
Status AppStatus `json:"status,omitempty"`
}

// AppSpec defines the desired state of App
type AppSpec struct {
DeploymentSpec DeploymentTemplate `json:"deploymentTemplate,omitempty"`
ServiceSpec ServiceTemplate `json:"serviceTemplate,omitempty"`
}

type DeploymentTemplate struct {
Name string `json:"name"`
Image string `json:"image"`
Replicas int32 `json:"replicas"`
}

type ServiceTemplate struct {
Name string `json:"name"`
}

// AppStatus defines the observed state of App.
// It should always be reconstructable from the state of the cluster and/or outside world.
type AppStatus struct {
DeploymentStatus *appsv1.DeploymentStatus `json:"deploymentStatus,omitempty"`
ServiceStatus *corev1.ServiceStatus `json:"serviceStatus,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// AppList contains a list of App
type AppList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []App `json:"items"`
}

使用controller-gen生成 deepcopy 文件

执行命令

1
2
cd crd-controller-demo
controller-gen object paths=pkg/apis/appcontroller/v1/types.go

命令执行结束后,在v1目录下生成了 zz_generated.deepcopy.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@master crd-controller-demo]# controller-gen object paths=pkg/apis/appcontroller/v1/types.go
[root@master crd-controller-demo]# tree
.
├── go.mod
├── go.sum
├── hack
│ ├── boilerplate.go.txt
│ ├── tools.go
│ └── update-codegen.sh
├── pkg
│ └── apis
│ └── appcontroller
│ ├── register.go
│ └── v1
│ ├── docs.go
│ ├── register.go
│ ├── types.go
│ └── zz_generated.deepcopy.go

使用code-generator生成 client,informer,lister

1、编写update-codegen.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#!/usr/bin/env bash

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# 设置脚本在执行过程中遇到任何错误时立即退出
set -o errexit
# 设置脚本在使用未定义的变量时立即退出
set -o nounset
# 设置脚本在管道命令中任意一条命令失败时立即退出
set -o pipefail

# 对generate-groups.sh 脚本的调用
../vendor/k8s.io/code-generator/generate-groups.sh \
"client,informer,lister" \
crd-controller-demo/pkg/generated \
crd-controller-demo/pkg/apis \
appcontroller:v1 \
--go-header-file $(pwd)/boilerplate.go.txt \
--output-base $(pwd)/../../

2、执行update-codegen.sh

最终会在pkg目录下,生成generated目录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
go mod tidy

# 生成vendor文件夹
go mod vendor

# 为vendor中的code-generator赋予权限
chmod -R 777 vendor

# 为hack中的update-codegen.sh脚本赋予权限
chmod -R 777 hack

# 调用脚本生成代码
$ cd hack && ./update-codegen.sh
Generating clientset for appcontroller:v1 at crd-controller-demo/pkg/generated/clientset
Generating listers for appcontroller:v1 at crd-controller-demo/pkg/generated/listers
Generating informers for appcontroller:v1 at crd-controller-demo/pkg/generated/informers

# 此时目录变为如下情况
$ cd .. && tree -L 5
.
├── go.mod
├── go.sum
├── hack
│ ├── boilerplate.go.txt
│ ├── tools.go
│ └── update-codegen.sh
├── pkg
│ ├── apis
│ │ └── appcontroller
│ │ ├── register.go
│ │ └── v1
│ │ ├── docs.go
│ │ ├── register.go
│ │ ├── types.go
│ │ └── zz_generated.deepcopy.go
│ └── generated
│ ├── clientset
│ │ └── versioned
│ │ ├── clientset.go
│ │ ├── fake
│ │ ├── scheme
│ │ └── typed
│ ├── informers
│ │ └── externalversions
│ │ ├── appcontroller
│ │ ├── factory.go
│ │ ├── generic.go
│ │ └── internalinterfaces
│ └── listers
│ └── appcontroller
│ └── v1
└── vendor
├── github.com

使用controller-gen生成crd

生成crd文件

1
2
cd crd-controller-demo
controller-gen crd paths=./... output:crd:dir=artifacts/crd

crd文件完整内容:记得在 annotations 中加入一个:api-approved.kubernetes.io: “https://github.com/kubernetes/kubernetes/pull/78458",否則kubernetes不让我们创建这个CRD

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: (devel)
api-approved.kubernetes.io: "https://github.com/kubernetes/kubernetes/pull/78458"
creationTimestamp: null
name: apps.appcontroller.k8s.io
spec:
group: appcontroller.k8s.io
names:
kind: App
listKind: AppList
plural: apps
singular: app
scope: Namespaced
versions:
- name: v1
schema:
openAPIV3Schema:
description: App is the Schema for the apps API
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: AppSpec defines the desired state of App
properties:
deploymentTemplate:
properties:
image:
type: string
name:
type: string
replicas:
format: int32
type: integer
required:
- image
- name
- replicas
type: object
serviceTemplate:
properties:
name:
type: string
required:
- name
type: object
type: object
status:
description: AppStatus defines the observed state of App. It should always
be reconstructable from the state of the cluster and/or outside world.
properties:
deploymentStatus:
description: DeploymentStatus is the most recently observed status
of the Deployment.
properties:
availableReplicas:
description: Total number of available pods (ready for at least
minReadySeconds) targeted by this deployment.
format: int32
type: integer
collisionCount:
description: Count of hash collisions for the Deployment. The
Deployment controller uses this field as a collision avoidance
mechanism when it needs to create the name for the newest ReplicaSet.
format: int32
type: integer
conditions:
description: Represents the latest available observations of a
deployment's current state.
items:
description: DeploymentCondition describes the state of a deployment
at a certain point.
properties:
lastTransitionTime:
description: Last time the condition transitioned from one
status to another.
format: date-time
type: string
lastUpdateTime:
description: The last time this condition was updated.
format: date-time
type: string
message:
description: A human readable message indicating details
about the transition.
type: string
reason:
description: The reason for the condition's last transition.
type: string
status:
description: Status of the condition, one of True, False,
Unknown.
type: string
type:
description: Type of deployment condition.
type: string
required:
- status
- type
type: object
type: array
observedGeneration:
description: The generation observed by the deployment controller.
format: int64
type: integer
readyReplicas:
description: readyReplicas is the number of pods targeted by this
Deployment with a Ready Condition.
format: int32
type: integer
replicas:
description: Total number of non-terminated pods targeted by this
deployment (their labels match the selector).
format: int32
type: integer
unavailableReplicas:
description: Total number of unavailable pods targeted by this
deployment. This is the total number of pods that are still
required for the deployment to have 100% available capacity.
They may either be pods that are running but not yet available
or pods that still have not been created.
format: int32
type: integer
updatedReplicas:
description: Total number of non-terminated pods targeted by this
deployment that have the desired template spec.
format: int32
type: integer
type: object
serviceStatus:
description: ServiceStatus represents the current status of a service.
properties:
conditions:
description: Current service state
items:
description: "Condition contains details for one aspect of the
current state of this API Resource. --- This struct is intended
for direct use as an array at the field path .status.conditions.
\ For example, \n type FooStatus struct{ // Represents the
observations of a foo's current state. // Known .status.conditions.type
are: \"Available\", \"Progressing\", and \"Degraded\" // +patchMergeKey=type
// +patchStrategy=merge // +listType=map // +listMapKey=type
Conditions []metav1.Condition `json:\"conditions,omitempty\"
patchStrategy:\"merge\" patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"`
\n // other fields }"
properties:
lastTransitionTime:
description: lastTransitionTime is the last time the condition
transitioned from one status to another. This should be
when the underlying condition changed. If that is not
known, then using the time when the API field changed
is acceptable.
format: date-time
type: string
message:
description: message is a human readable message indicating
details about the transition. This may be an empty string.
maxLength: 32768
type: string
observedGeneration:
description: observedGeneration represents the .metadata.generation
that the condition was set based upon. For instance, if
.metadata.generation is currently 12, but the .status.conditions[x].observedGeneration
is 9, the condition is out of date with respect to the
current state of the instance.
format: int64
minimum: 0
type: integer
reason:
description: reason contains a programmatic identifier indicating
the reason for the condition's last transition. Producers
of specific condition types may define expected values
and meanings for this field, and whether the values are
considered a guaranteed API. The value should be a CamelCase
string. This field may not be empty.
maxLength: 1024
minLength: 1
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
type: string
status:
description: status of the condition, one of True, False,
Unknown.
enum:
- "True"
- "False"
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
--- Many .condition.type values are consistent across
resources like Available, but because arbitrary conditions
can be useful (see .node.status.conditions), the ability
to deconflict is important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
required:
- lastTransitionTime
- message
- reason
- status
- type
type: object
type: array
x-kubernetes-list-map-keys:
- type
x-kubernetes-list-type: map
loadBalancer:
description: LoadBalancer contains the current status of the load-balancer,
if one is present.
properties:
ingress:
description: Ingress is a list containing ingress points for
the load-balancer. Traffic intended for the service should
be sent to these ingress points.
items:
description: 'LoadBalancerIngress represents the status
of a load-balancer ingress point: traffic intended for
the service should be sent to an ingress point.'
properties:
hostname:
description: Hostname is set for load-balancer ingress
points that are DNS based (typically AWS load-balancers)
type: string
ip:
description: IP is set for load-balancer ingress points
that are IP based (typically GCE or OpenStack load-balancers)
type: string
ipMode:
description: IPMode specifies how the load-balancer
IP behaves, and may only be specified when the ip
field is specified. Setting this to "VIP" indicates
that traffic is delivered to the node with the destination
set to the load-balancer's IP and port. Setting this
to "Proxy" indicates that traffic is delivered to
the node or pod with the destination set to the node's
IP and node port or the pod's IP and port. Service
implementations may use this information to adjust
traffic routing.
type: string
ports:
description: Ports is a list of records of service ports
If used, every port defined in the service should
have an entry in it
items:
properties:
error:
description: 'Error is to record the problem with
the service port The format of the error shall
comply with the following rules: - built-in
error values shall be specified in this file
and those shall use CamelCase names - cloud
provider specific error values must have names
that comply with the format foo.example.com/CamelCase.
--- The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)'
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
port:
description: Port is the port number of the service
port of which status is recorded here
format: int32
type: integer
protocol:
default: TCP
description: 'Protocol is the protocol of the
service port of which status is recorded here
The supported values are: "TCP", "UDP", "SCTP"'
type: string
required:
- port
- protocol
type: object
type: array
x-kubernetes-list-type: atomic
type: object
type: array
type: object
type: object
type: object
type: object
served: true
storage: true

手动注册版本v1的 CRD 资源

  • 在生成了客户端代码后,我们还需要手动注册版本v1的CRD资源,才能真正使用这个client,不然在编译时会出现 undefined: v1alpha1.AddToScheme 错误、undefined: v1alpha1.Resource 错误。
  • v1alpha1.AddToScheme、v1alpha1.Resource 这两个是用于 client 注册的

编写 v1/register.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
package v1

import (
"crd-controller-demo/pkg/apis/appcontroller"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
)

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{Group: appcontroller.GroupName, Version: "v1"}

// Kind takes an unqualified kind and returns back a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
return SchemeGroupVersion.WithResource(resource).GroupResource()
}

var (
// SchemeBuilder initializes a scheme builder
SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
// AddToScheme is a global function that registers this API group & version to a scheme
AddToScheme = SchemeBuilder.AddToScheme
)

// Adds the list of known types to Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(SchemeGroupVersion,
&App{},
&AppList{},
)
metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
return nil
}