前言
之前部署了 Kubernetes 1.13.0, 发现 master 服务的启动方式与 1.10.4 版本有所区别, kube-apiserver,kube-controller-manager 和 kube-scheduler 分别使用不同的镜像启动, 而不再是公用一个 hyperkube 镜像. 但是官方的 kube-controller-manager 镜像中不包含 ceph client, 导致无法创建 RBD volume. 于是需要打包自定义镜像, 安装 ceph client.
1. 环境
系统: CentOS 7.2
- Docker:18.03.1-ce
- Kubernetes:1.13.0
2. 下载 Kubernetes 源码
使用 Git clone -b 参数下载对应版本的源码:
# Git clone -b v1.13.0 https://github.com/kubernetes/kubernetes.git
3. 下载 base 镜像
打包过程使用到的 base 镜像有:
- k8s.gcr.io/kube-cross:v1.11.2-1
- k8s.gcr.io/pause-amd64:3.1
- k8s.gcr.io/debian-base-amd64:0.4.0
- k8s.gcr.io/debian-iptables-amd64:v11.0
- k8s.gcr.io/debian-hyperkube-base-amd64:0.12.0
这些镜像可以使用 make release 命令从源码编译生成, 为了方便, 我直接使用官方编译好的镜像. 我将它们下载下来上传到了私有镜像仓库, 方便在不同机器上打包.
4. 修改镜像地址
由于使用本地镜像仓库, 要修改打包脚本中的镜像仓库地址. 修改如下:
- build/build-image/Dockerfile:
- # Git diff build/build-image/Dockerfile
- diff --Git a/build/build-image/Dockerfile b/build/build-image/Dockerfile
- index ff4543b..976a377 100644
- --- a/build/build-image/Dockerfile
- +++ b/build/build-image/Dockerfile
- @@ -13,7 +13,7 @@
- # limitations under the License.
- # This file creates a standard build environment for building Kubernetes
- - FROM k8s.gcr.io/kube-cross:KUBE_BUILD_IMAGE_CROSS_TAG
- +FROM registry.example.com/k8s_gcr/kube-cross:KUBE_BUILD_IMAGE_CROSS_TAG
- # Mark this as a kube-build container
- RUN touch /kube-build-image
- build/common.sh:
- # Git diff build/common.sh
- diff --Git a/build/common.sh b/build/common.sh
- index b3b7748..902b08f 100755
- --- a/build/common.sh
- +++ b/build/common.sh
- @@ -91,14 +91,15 @@ kube::build::get_docker_wrapped_binaries() {
- local arch=$1
- local debian_base_version=0.4.0
- local debian_iptables_version=v11.0
- + local registry="registry.example.com/k8s_gcr"
- ### If you change any of these lists, please also update DOCKERIZED_BINARIES
- ### in build/BUILD. And kube::golang::server_image_targets
- local targets=(
- - cloud-controller-manager,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"
- - kube-apiserver,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"
- - kube-controller-manager,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"
- - kube-scheduler,"k8s.gcr.io/debian-base-${arch}:${debian_base_version}"
- - kube-proxy,"k8s.gcr.io/debian-iptables-${arch}:${debian_iptables_version}"
- + cloud-controller-manager,"${registry}/debian-base-${arch}:${debian_base_version}"
- + kube-apiserver,"${registry}/debian-base-${arch}:${debian_base_version}"
- + kube-controller-manager,"${registry}/debian-base-${arch}:${debian_base_version}"
- + kube-scheduler,"${registry}/debian-base-${arch}:${debian_base_version}"
- + kube-proxy,"${registry}/debian-iptables-${arch}:${debian_iptables_version}"
- )
- echo "${targets[@]}"
- build/root/WORKSPACE:
- # Git diff build/root/WORKSPACE
- diff --Git a/build/root/WORKSPACE b/build/root/WORKSPACE
- index cee8962..f1a7c37 100644
- --- a/build/root/WORKSPACE
- +++ b/build/root/WORKSPACE
- @@ -71,7 +71,7 @@ http_file(
- docker_pull(
- name = "debian-base-amd64",
- digest = "sha256:86176bc8ccdc4d8ea7fbf6ba4b57fcefc2cb61ff7413114630940474ff9bf751",
- - registry = "k8s.gcr.io",
- + registry = "registry.example.com/k8s_gcr",
- repository = "debian-base-amd64",
- tag = "0.4.0", # ignored, but kept here for documentation
- )
- @@ -79,7 +79,7 @@ docker_pull(
- docker_pull(
- name = "debian-iptables-amd64",
- digest = "sha256:d4ff8136b9037694a3165a7fff6a91e7fc828741b8ea1eda226d4d9ea5d23abb",
- - registry = "k8s.gcr.io",
- + registry = "registry.example.com/k8s_gcr",
- repository = "debian-iptables-amd64",
- tag = "v11.0", # ignored, but kept here for documentation
- )
- @@ -87,7 +87,7 @@ docker_pull(
- docker_pull(
- name = "debian-hyperkube-base-amd64",
- digest = "sha256:4a77bc882f7d629c088a11ff144a2e86660268fddf63b61f52b6a93d16ab83f0",
- - registry = "k8s.gcr.io",
- + registry = "registry.example.com/k8s_gcr",
- repository = "debian-hyperkube-base-amd64",
- tag = "0.12.0", # ignored, but kept here for documentation
- )
- build/lib/release.sh:
- # Git diff build/lib/release.sh
- diff --Git a/build/lib/release.sh b/build/lib/release.sh
- index d7ccc01..47d9e37 100644
- --- a/build/lib/release.sh
- +++ b/build/lib/release.sh
- @@ -327,7 +327,7 @@ function kube::release::create_docker_images_for_server() {
- local images_dir="${RELEASE_IMAGES}/${arch}"
- mkdir -p "${images_dir}"
- - local -r docker_registry="k8s.gcr.io"
- + local -r docker_registry="registry.example.com/k8s_gcr"
- # Docker tags cannot contain '+'
- local docker_tag="${KUBE_GIT_VERSION/+/_}"
- if [[ -z "${docker_tag}" ]]; then
5. kube-controller-manager 安装 ceph client
Kubernetes master 服务镜像的 build 在 build/lib/release.sh 的 kube::release::create_docker_images_for_server() 中进行, 它们的 Dockerfile 也是由这个函数动态生成的. 修改这个函数, 在 Dockerfile 中增加上安装 ceph client 的命令.
由于使用的 base 镜像 k8s.gcr.io/debian-base-amd64:0.4.0 是 Debian 9.5 Stretch 版本的, ceph mimic 使用 C++ 17 标准, 需要 gcc 8 支持, 所以不支持 Debian Stretch. 于是只能安装 ceph luminous 版本的 client. 当然也可以使用第三方编译的 deb 包, 比如.
在步骤四的修改上继续修改 build/lib/release.sh:
- # Git diff build/lib/release.sh
- diff --Git a/build/lib/release.sh b/build/lib/release.sh
- index d7ccc01..0d03da9 100644
- --- a/build/lib/release.sh
- +++ b/build/lib/release.sh
- @@ -327,7 +327,7 @@ function kube::release::create_docker_images_for_server() {
- local images_dir="${RELEASE_IMAGES}/${arch}"
- mkdir -p "${images_dir}"
- - local -r docker_registry="k8s.gcr.io"
- + local -r docker_registry="registry.example.com/k8s_gcr"
- # Docker tags cannot contain '+'
- local docker_tag="${KUBE_GIT_VERSION/+/_}"
- if [[ -z "${docker_tag}" ]]; then
- @@ -370,11 +370,21 @@ function kube::release::create_docker_images_for_server() {
- cat <<EOF> "${docker_file_path}"
- FROM ${base_image}
- COPY ${binary_name} /usr/local/bin/${binary_name}
- +RUN echo "deb http://mirrors.aliyun.com/debian/ stretch main non-free contrib\ndeb-src http://mirrors.aliyun.com/debian/ stretch main non-free contrib\
- +RUN apt-get update && apt-get -y install apt-transport-https gnupg2 wget curl
- EOF
- # ensure /etc/nsswitch.conf exists so go's resolver respects /etc/hosts
- if [[ "${base_image}" =~ busybox ]]; then
- echo "COPY nsswitch.conf /etc/">> "${docker_file_path}"
- fi
- +
- + # install ceph client
- + if [[ ${binary_name} =~ "kube-controller-manager" ]]; then
- + echo "RUN wget -q -O-'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc'| apt-key add -">> "${docker_file_path}"
- + echo "RUN echo'deb https://download.ceph.com/debian-luminous/ stretch main'> /etc/apt/sources.list.d/ceph.list">> "${docker_file_path}"
- + echo "RUN apt-get update && apt-get install -y ceph-common ceph-fuse">> "${docker_file_path}"
- + fi
- +
- "${DOCKER[@]}" build -q -t "${docker_image_tag}" "${docker_build_path}">/dev/null
- "${DOCKER[@]}" save "${docker_image_tag}"> "${binary_dir}/${binary_name}.tar"
- echo "${docker_tag}"> "${binary_dir}/${binary_name}.docker_tag"
将所有镜像的源换成了 aliyun, 并在 kube-controller-manager Dockerfile 中增加了安装 ceph client 的命令.
至此准备工作就完成了, 下面开始编译镜像.
6. 编译 kubernetes 镜像
编译命令:
- # cd kubernetes
- # make clean
- # KUBE_BUILD_PLATFORMS=Linux/amd64 KUBE_BUILD_CONFORMANCE=n KUBE_BUILD_HYPERKUBE=n make release-images GOFLAGS=-v GOGCFLAGS="-N -l"
其中 KUBE_BUILD_PLATFORMS=Linux/amd64 指定目标平台为 Linux/amd64,GOFLAGS=-v 开启 verbose 日志, GOGCFLAGS="-N -l" 禁止编译优化和内联, 减小可执行程序大小.
7. 导入镜像
等待编译完成后, 在_output/release-stage/server/Linux-amd64/kubernetes/server/bin / 目录下保存了编译生成的二进制可执行程序和 docker 镜像 tar 包. 导入 kube-controller-manager 镜像, 上传到镜像仓库:
- # docker load -i _output/release-stage/server/Linux-amd64/kubernetes/server/bin/kube-controller-manager.tar
- # docker tag registry.example.com/k8s_gcr/kube-controller-manager:v1.13.0 registry.example.com/k8s_gcr/kube-controller-manager:v1.13.0-ceph-mimic
- # docker push registry.example.com/k8s_gcr/kube-controller-manager:v1.13.0-ceph-mimic
整个编译过程结束, 现在就可以到 master 节点上, 修改 / etc/kubernetes/manifests/kube-controller-manager.YAML 描述文件中的 image, 修改完立即生效, 创建 RBD PV 可以看到效果.
测试发现 kube-controller-manager 创建 rbd 没问题, 但是 node 节点挂载 rbd 时报错. 进一步测试发现与 ceph client 版本的关系不大, 而与 kernel 的版本有关. 将 node 节点的内核从 3.10 升级到 4.17 后, 可以正常挂载.
参考资料
croit | Debian 9 (Stretch) Ceph Mimic mirror
来源: https://www.cnblogs.com/ltxdzh/p/10137786.html