一, 安装 Jenkins
1. 安装存储服务器
找一台服务器搭建一台 nfs 服务器
系统: Ubuntu 16.04
- IP:172.18.1.13
- apt install nfs-common nfs-kernel-server -y
- # 配置挂载信息
- cat /etc/exports
- /data/k8s *(rw,sync,no_root_squash)
- # 给目录添加权限
- chmod -R 777 /data/k8s
- # 启动
- /etc/init.d/nfs-kernel-server start
- # 开机启动
- systemctl enable nfs-kernel-server
2. kubernetes 集群安装 Jenkins
- # 该目录下是 jenkins.mytest.io 为测试域名的自签密钥
- ls jenkins.mytest.io/
- cacerts.pem cacerts.srl cakey.pem create_self-signed-cert.sh jenkins.mytest.io.crt jenkins.mytest.io.csr jenkins.mytest.io.key openssl.cnf tls.crt tls.key
- ---
- cd jenkins.mytest.io
- # 创建 Jenkins 所在的 namespace
- kubectl create namespace kube-ops
- # 将密文添加到 kube-ops 里面
- # 服务证书和私钥密文
- kubectl -n kube-ops create secret tls tls-jenkins-ingress --cert=./tls.crt --key=./tls.key
- # ca 证书密文
- kubectl -n kube-ops create secret generic tls-ca --from-file=cacerts.pem
3. 创建 Jenkins
- cat jenkins-pvc.YAML
- ---
- apiVersion: v1
- kind: PersistentVolume
- metadata:
- name: jenkins-pv
- spec:
- capacity:
- storage: 20Gi
- accessModes:
- - ReadWriteMany
- persistentVolumeReclaimPolicy: Delete
- nfs:
- server: 172.18.1.13
- path: /data/k8s
- ---
- kind: PersistentVolumeClaim
- apiVersion: v1
- metadata:
- name: jenkins-pvc
- namespace: kube-ops
- spec:
- accessModes:
- - ReadWriteMany
- resources:
- requests:
- storage: 20Gi
cat rbac.YAML
- ---
- apiVersion: v1
- kind: ServiceAccount
- metadata:
- name: jenkins
- namespace: kube-ops
- ---
- kind: ClusterRole
- apiVersion: rbac.authorization.k8s.io/v1beta1
- metadata:
- name: jenkins
- rules:
- - apiGroups: ["extensions", "apps"]
- resources: ["deployments"]
- verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- - apiGroups: [""]
- resources: ["services"]
- verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
- - apiGroups: [""]
- resources: ["pods"]
- verbs: ["create","delete","get","list","patch","update","watch"]
- - apiGroups: [""]
- resources: ["pods/exec"]
- verbs: ["create","delete","get","list","patch","update","watch"]
- - apiGroups: [""]
- resources: ["pods/log"]
- verbs: ["get","list","watch"]
- - apiGroups: [""]
- resources: ["secrets"]
- verbs: ["get"]
- ---
- apiVersion: rbac.authorization.k8s.io/v1beta1
- kind: ClusterRoleBinding
- metadata:
- name: jenkins
- namespace: kube-ops
- roleRef:
- apiGroup: rbac.authorization.k8s.io
- kind: ClusterRole
- name: jenkins
- subjects:
- - kind: ServiceAccount
- name: jenkins
- namespace: kube-ops
cat jenkins.YAML
- ---
- apiVersion: extensions/v1beta1
- kind: Deployment
- metadata:
- name: jenkins
- namespace: kube-ops
- spec:
- template:
- metadata:
- labels:
- App: jenkins
- spec:
- terminationGracePeriodSeconds: 10
- serviceAccount: jenkins
- containers:
- - name: jenkins
- image: jenkins/jenkins:lts
- imagePullPolicy: IfNotPresent
- ports:
- - containerPort: 8080
- name: web
- protocol: TCP
- - containerPort: 50000
- name: agent
- protocol: TCP
- resources:
- limits:
- CPU: 1000m
- memory: 1Gi
- requests:
- CPU: 500m
- memory: 512Mi
- livenessProbe:
- httpGet:
- path: /login
- port: 8080
- initialDelaySeconds: 60
- timeoutSeconds: 5
- failureThreshold: 12
- readinessProbe:
- httpGet:
- path: /login
- port: 8080
- initialDelaySeconds: 60
- timeoutSeconds: 5
- failureThreshold: 12
- volumeMounts:
- - name: jenkinshome
- subPath: jenkins
- mountPath: /var/jenkins_home
- env:
- - name: LIMITS_MEMORY
- valueFrom:
- resourceFieldRef:
- resource: limits.memory
- divisor: 1Mi
- - name: JAVA_OPTS
- value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
- securityContext:
- fsGroup: 1000
- volumes:
- - name: jenkinshome
- persistentVolumeClaim:
- claimName: jenkins-pvc
- ---
- apiVersion: v1
- kind: Service
- metadata:
- name: jenkins
- namespace: kube-ops
- labels:
- App: jenkins
- spec:
- selector:
- App: jenkins
- type: NodePort
- ports:
- - name: Web
- port: 8080
- targetPort: Web
- nodePort: 30002
- - name: agent
- port: 50000
- targetPort: agent
- ---
- apiVersion: extensions/v1beta1
- kind: Ingress
- metadata:
- name: jenkins-lb
- namespace: kube-ops
- spec:
- tls:
- - secretName: tls-jenkins-ingress
- rules:
- - host: jenkins.mytest.io
- http:
- paths:
- - backend:
- serviceName: jenkins
- servicePort: 8080
创建 Jenkins
- kubectl create -f pvc.YAML
- kubectl create -f rbac.YAML
- kubectl create -f jenkins.YAML
二, jenkins 配置
在 / etc/hosts 配置域名解析
kube-ip jenkins.mytest.io
1. 初始化配置
打开 https://jenkins.mytes.io
安装插件, 选择默认即可
2. 插件配置
采用 Jenkins 里面的 kubernetes 插件, 让 Jenkins 可以调用 kubernetes 生成 Jenkins-slave
https://github.com/jenkinsci/kubernetes-plugin https://github.com/jenkinsci/kubernetes-plugin
2.1 安装 kubernetes 插件
Manage Jenkins -> Manage Plugins -> Available -> Kubernetes plugin 勾选安装即可.
2.2 配置 kubernetes 插件功能
Manage Jenkins -> Configure System -> (拖到最下方)Add a new cloud -> 选择 Kubernetes, 然后填写 Kubernetes 和 Jenkins 配置信息.
kubernetes 地址采用了 kube 的服务器发现 https://kubernetes.default.svc.cluster.local
namespace 填 kube-ops, 然后点击 Test Connection, 如果出现 Connection test successful 的提示信息证明 Jenkins 已经可以和 Kubernetes 系统正常通信
Jenkins URL 地址: http://jenkins.kube-ops.svc.cluster.local:8080/
2.3 配置 kubernetes Pod Template
其实就是配置 Jenkins Slave 运行的 Pod 模板, 命名空间我们同样是用 kube-ops,Labels 这里也非常重要, 对于后面执行 Job 的时候需要用到该值, 然后我们这里使用的是 cnych/jenkins:jnlp 这个镜像, 这个镜像是在官方的 jnlp 镜像基础上定制的, 加入了 kubectl 等一些实用的工具.
2.4 添加容器的挂载卷
另外需要注意我们这里需要在下面挂载两个主机目录, 一个是 /var/run/docker.sock, 该文件是用于 Pod 中的容器能够共享宿主机的 Docker, 这就是大家说的 docker in docker 的方式, Docker 二进制文件我们已经打包到上面的镜像中了, 另外一个目录下 /root/.kube 目录, 我们将这个目录挂载到容器的 /home/jenkins/.kube 目录下面这是为了让我们能够在 Pod 的容器中能够使用 kubectl 工具来访问我们的 Kubernetes 集群, 方便我们后面在 Slave Pod 部署 Kubernetes 应用.
2.5 添加账号
另外一些同学在配置了后运行 Slave Pod 的时候出现了权限问题, 因为 Jenkins Slave Pod 中没有配置权限, 所以需要配置上 ServiceAccount, 在 Slave Pod 配置的地方点击下面的高级, 添加上对应的 ServiceAccount 即可:
测试的时候不添加账号会告知没有权限
在容器模板高级里面添加 kubernetes 集群中创建的 jenkins 账号
三, 测试
创建一个测试任务
在 pipeline 的框里面添加一下内容
- def label = "jnlp-slave"
- podTemplate(inheritFrom: 'jnlp-slave', instanceCap: 0, label: 'jnlp-slave', name: '', namespace:'kube-ops', nodeSelector:'', podRetention: always(), serviceAccount: '', workspaceVolume: emptyDirWorkspaceVolume(false), YAML:'') {
- node(label) {
- container('jnlp-slave'){
- stage('Run shell') {
- sh 'docker info'
- sh 'kubectl get pods -n kube-ops'
- }
- }
- }
- }
开始构建任务
构建任务输出
- Started by user admin
- Running in Durability level: MAX_SURVIVABILITY
- [Pipeline] Start of Pipeline
- [Pipeline] podTemplate
- [Pipeline] {
- [Pipeline] node
- Still waiting to schedule task
- 'Jenkins' doesn't have label'jnlp-slave'
- Agent jnlp-slave-tbdnl is provisioned from template Kubernetes Pod Template
- Agent specification [Kubernetes Pod Template] (jnlp-slave):
- * [jnlp-slave] cnych/jenkins:jnlp
- Running on jnlp-slave-tbdnl in /home/jenkins/workspace/test-jnlp-slave
- [Pipeline] {
- [Pipeline] container
- [Pipeline] {
- [Pipeline] stage
- [Pipeline] { (Run shell)
- [Pipeline] sh
- + docker info
- Containers: 15
- Running: 12
- Paused: 0
- Stopped: 3
- Images: 12
- Server Version: 18.09.6
- Storage Driver: overlay2
- Backing Filesystem: extfs
- Supports d_type: true
- Native Overlay Diff: false
- Logging Driver: JSON-file
- Cgroup Driver: cgroupfs
- Plugins:
- Volume: local
- Network: bridge host macvlan null overlay
- Log: awslogs Fluentd gcplogs gelf journald JSON-file local logentries splunk syslog
- Swarm: inactive
- Runtimes: runc
- Default Runtime: runc
- Init Binary: docker-init
- containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
- runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
- init version: fec3683
- Security Options:
- apparmor
- seccomp
- Profile: default
- Kernel Version: 4.15.0-1049-azure
- Operating System: Ubuntu 16.04.6 LTS
- OSType: Linux
- Architecture: x86_64
- CPUs: 2
- Total Memory: 7.768GiB
- Name: test-kube-node-04
- ID: YFTJ:FVHK:TAF3:HTAJ:HJ2A:5SFW:73RW:VQY5:Y64U:UGIR:KMJ2:XPRL
- Docker Root Dir: /var/lib/docker
- Debug Mode (client): false
- Debug Mode (server): false
- Registry: https://index.docker.io/v1/
- Labels:
- Experimental: false
- Insecure Registries:
- 127.0.0.0/8
- Registry Mirrors:
- https://kv3qfp85.mirror.aliyuncs.com/
- Live Restore Enabled: false
- WARNING: No swap limit support
- [Pipeline] sh
- + kubectl get pods -n kube-ops
- NAME READY STATUS RESTARTS AGE
- jenkins-6b874b8d7-q28h4 1/1 Running 0 3h
- jnlp-slave-tbdnl 2/2 Running 0 15s
- [Pipeline] }
- [Pipeline] // stage
- [Pipeline] }
- [Pipeline] // container
- [Pipeline] }
- [Pipeline] // node
- [Pipeline] }
- [Pipeline] // podTemplate
- [Pipeline] End of Pipeline
- Finished: SUCCESS
结束语
上面步骤大部分都是根据阳明博客上的《基于 Jenkins 的 CI/CD(一)》思路跟进, 但是由于环境和自己认知问题, 会出现各种出错, 憋了一天, 没什么进展.
后来只能根据 kubectl -n kube-ops logs -f jenkins-xxxxx 的命令一点点查出来的, 搜过很多帖子大概思路一致, 但是无法解决本质问题, 如果跑不起来, 再高端也是个没有用, 后来根据 kubernetes-plugin 的 GitHub, 具体读了遍结合自己报的错一点一点调整过来, 总算搞出来了.
具体参考了以下几遍优秀的文章:
基于 Jenkins 的 CI/CD(一)(阳明老师的文章很棒)
https://www.qikqiak.com/post/kubernetes-jenkins1
kubernetes Jenkins GitLab 搭建 CI/CD 环境 (二)
GitHub-Jenkins-kubernetes-plugin:jenkinsci/kubernetes-plugin https://github.com/jenkinsci/kubernetes-plugin
rancher 官网: 在 Kubernetes 上部署和扩展 Jenkins
来源: http://www.bubuko.com/infodetail-3112719.html