十一、安装kubesphere3.3

官网参考文档

https://kubesphere.com.cn/docs/v3.3/pluggable-components/alerting/

1.部署kubesphere时需要默认 StorageClass

kubectl edit sc nfs-boge

  metadata:annotations:storageclass.beta.kubernetes.io/is-default-class: "true"

2.下载yaml

wget https://github.com/kubesphere/ks-installer/releases/download/v3.3.0/kubesphere-installer.yamlwget https://github.com/kubesphere/ks-installer/releases/download/v3.3.0/cluster-configuration.yaml#修改cluster-configuration.yaml
#将ectd下的 endpointIps改为你的master节点的私有IP地址。
#endpointIps: XX.X.X.X

3.运行yaml

kubectl apply -f kubesphere-installer.yamlkubectl apply -f cluster-configuration.yaml

4. 查看日志

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f#访问任意机器的 30880端口
#账号 ： admin
#密码 ： P@88w0rd

5. 解决etcd监控证书找不到问题

kubectl -n kubesphere-monitoring-system create secret generic kube-etcd-client-certs  \
--from-file=etcd-client-ca.crt=/etc/kubernetes/pki/etcd/ca.crt  \
--from-file=etcd-client.crt=/etc/kubernetes/pki/etcd/healthcheck-client.crt  \
--from-file=etcd-client.key=/etc/kubernetes/pki/etcd/healthcheck-client.key

6. 在安装后启用告警系统,

在 cluster-configuration.yaml 文件中，搜索 alerting，将 enabled 的 false 更改为 true 以启用告警系统。完成后保存文件

alerting:enabled: true # 将“false”更改为“true”。#运行
kubectl apply -f kubesphere-installer.yamlkubectl apply -f cluster-configuration.yaml

6.0 配置钉钉报警

6.1 钉钉自定义机器配置

添加自定义机器人，安全配置，勾选** 加签**

6.2 操作步骤

左上角>平台管理>平台设置>通知管理>通知配置>钉钉>群机器人配置
开启-已启用
填写自己的 Webhook URL
填写自己的 密钥 （加签）
发送测试信息
确定

查看钉钉群消息.是否发送成功？？？？？

7. 在安装后启用应用商店，

在该 YAML 文件中，搜索 openpitrix，将 enabled 的 false 改为 true。完成后，点击右下角的确定，保存配置。

openpitrix:store:enabled: true # 将“false”更改为“true”。#运行
kubectl apply -f kubesphere-installer.yamlkubectl apply -f cluster-configuration.yaml

8.在安装后启用服务网格 istio

在该配置文件中，搜索 servicemesh，并将 enabled 的 false 改为 true。完成后，点击右下角的确定，保存配置

servicemesh:
enabled: true # 将“false”更改为“true”。
istio: # Customizing the istio installation configuration, refer to https://istio.io/latest/docs/setup/additional-setup/customize-installation/components:ingressGateways:- name: istio-ingressgateway # 将服务暴露至服务网格之外。默认不开启。enabled: falsecni:enabled: false # 启用后，会在 Kubernetes pod 生命周期的网络设置阶段完成 Istio 网格的 pod 流量转发设置工作。

9.在安装前启用 DevOps

在该 YAML 文件中，搜索 devops，将 enabled 的 false 改为 true。完成后，点击右下角的确定，保存配置。

devops:enabled: true # 将“false”更改为“true”。

10. 卸载方法

kubectl delete -f cluster-configuration.yaml --force
kubectl delete -f kubesphere-installer.yaml --force

#删除残余文件
vi del.sh

#!/usr/bin/env bashfunction delete_sure(){cat << eof
$(echo -e "\033[1;36mNote:\033[0m")
Delete the KubeSphere cluster, including the module kubesphere-system kubesphere-devops-system kubesphere-devops-worker kubesphere-monitoring-system kubesphere-logging-system openpitrix-system.
eofread -p "Please reconfirm that you want to delete the KubeSphere cluster.  (yes/no) " ans
while [[ "x"$ans != "xyes" && "x"$ans != "xno" ]]; doread -p "Please reconfirm that you want to delete the KubeSphere cluster.  (yes/no) " ans
doneif [[ "x"$ans == "xno" ]]; thenexit
fi
}delete_sure# delete ks-installer
kubectl delete deploy ks-installer -n kubesphere-system 2>/dev/null# delete helm
for namespaces in kubesphere-system kubesphere-devops-system kubesphere-monitoring-system kubesphere-logging-system openpitrix-system kubesphere-monitoring-federated
dohelm list -n $namespaces | grep -v NAME | awk '{print $1}' | sort -u | xargs -r -L1 helm uninstall -n $namespaces 2>/dev/null
done# delete kubefed
kubectl get cc -n kubesphere-system ks-installer -o jsonpath="{.status.multicluster}" | grep enable
if [[ $? -eq 0 ]]; then# delete kubefed types resourcesfor kubefed in `kubectl api-resources --namespaced=true --api-group=types.kubefed.io -o name`dokubectl delete -n kube-federation-system $kubefed --all 2>/dev/nulldonefor kubefed in `kubectl api-resources --namespaced=false --api-group=types.kubefed.io -o name`dokubectl delete $kubefed --all 2>/dev/nulldone# delete kubefed core resoucesfor kubefed in `kubectl api-resources --namespaced=true --api-group=core.kubefed.io -o name`dokubectl delete -n kube-federation-system $kubefed --all 2>/dev/nulldonefor kubefed in `kubectl api-resources --namespaced=false --api-group=core.kubefed.io -o name`dokubectl delete $kubefed --all 2>/dev/nulldone# uninstall kubefed charthelm uninstall -n kube-federation-system kubefed 2>/dev/null
fihelm uninstall -n kube-system snapshot-controller 2>/dev/null# delete kubesphere deployment & statefulset
kubectl delete deployment -n kubesphere-system `kubectl get deployment -n kubesphere-system -o jsonpath="{.items[*].metadata.name}"` 2>/dev/null
kubectl delete statefulset -n kubesphere-system `kubectl get statefulset -n kubesphere-system -o jsonpath="{.items[*].metadata.name}"` 2>/dev/null# delete monitor resources
kubectl delete prometheus -n kubesphere-monitoring-system k8s 2>/dev/null
kubectl delete Alertmanager -n kubesphere-monitoring-system main 2>/dev/null
kubectl delete DaemonSet -n kubesphere-monitoring-system node-exporter 2>/dev/null
kubectl delete statefulset -n kubesphere-monitoring-system `kubectl get statefulset -n kubesphere-monitoring-system -o jsonpath="{.items[*].metadata.name}"` 2>/dev/null# delete grafana
kubectl delete deployment -n kubesphere-monitoring-system grafana 2>/dev/null
kubectl --no-headers=true get pvc -n kubesphere-monitoring-system -o custom-columns=:metadata.namespace,:metadata.name | grep -E kubesphere-monitoring-system | xargs -n2 kubectl delete pvc -n 2>/dev/null# delete pvc
pvcs="kubesphere-system|openpitrix-system|kubesphere-devops-system|kubesphere-logging-system"
kubectl --no-headers=true get pvc --all-namespaces -o custom-columns=:metadata.namespace,:metadata.name | grep -E $pvcs | xargs -n2 kubectl delete pvc -n 2>/dev/null# delete rolebindings
delete_role_bindings() {for rolebinding in `kubectl -n $1 get rolebindings -l iam.kubesphere.io/user-ref -o jsonpath="{.items[*].metadata.name}"`dokubectl -n $1 delete rolebinding $rolebinding 2>/dev/nulldone
}# delete roles
delete_roles() {kubectl -n $1 delete role admin 2>/dev/nullkubectl -n $1 delete role operator 2>/dev/nullkubectl -n $1 delete role viewer 2>/dev/nullfor role in `kubectl -n $1 get roles -l iam.kubesphere.io/role-template -o jsonpath="{.items[*].metadata.name}"`dokubectl -n $1 delete role $role 2>/dev/nulldone
}# remove useless labels and finalizers
for ns in `kubectl get ns -o jsonpath="{.items[*].metadata.name}"`
dokubectl label ns $ns kubesphere.io/workspace-kubectl label ns $ns kubesphere.io/namespace-kubectl patch ns $ns -p '{"metadata":{"finalizers":null,"ownerReferences":null}}'delete_role_bindings $nsdelete_roles $ns
done# delete clusterroles
delete_cluster_roles() {for role in `kubectl get clusterrole -l iam.kubesphere.io/role-template -o jsonpath="{.items[*].metadata.name}"`dokubectl delete clusterrole $role 2>/dev/nulldonefor role in `kubectl get clusterroles | grep "kubesphere" | awk '{print $1}'| paste -sd " "`dokubectl delete clusterrole $role 2>/dev/nulldone
}
delete_cluster_roles# delete clusterrolebindings
delete_cluster_role_bindings() {for rolebinding in `kubectl get clusterrolebindings -l iam.kubesphere.io/role-template -o jsonpath="{.items[*].metadata.name}"`dokubectl delete clusterrolebindings $rolebinding 2>/dev/nulldonefor rolebinding in `kubectl get clusterrolebindings | grep "kubesphere" | awk '{print $1}'| paste -sd " "`dokubectl delete clusterrolebindings $rolebinding 2>/dev/nulldone
}
delete_cluster_role_bindings# delete clusters
for cluster in `kubectl get clusters -o jsonpath="{.items[*].metadata.name}"`
dokubectl patch cluster $cluster -p '{"metadata":{"finalizers":null}}' --type=merge
done
kubectl delete clusters --all 2>/dev/null# delete workspaces
for ws in `kubectl get workspaces -o jsonpath="{.items[*].metadata.name}"`
dokubectl patch workspace $ws -p '{"metadata":{"finalizers":null}}' --type=merge
done
kubectl delete workspaces --all 2>/dev/null# make DevOps CRs deletable
for devops_crd in $(kubectl get crd -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep "devops.kubesphere.io"); dofor ns in $(kubectl get ns -ojsonpath='{.items..metadata.name}'); dofor devops_res in $(kubectl get $devops_crd -n $ns -oname); dokubectl patch $devops_res -n $ns -p '{"metadata":{"finalizers":[]}}' --type=mergedonedone
done# delete validatingwebhookconfigurations
for webhook in ks-events-admission-validate users.iam.kubesphere.io network.kubesphere.io validating-webhook-configuration resourcesquotas.quota.kubesphere.io
dokubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io $webhook 2>/dev/null
done# delete mutatingwebhookconfigurations
for webhook in ks-events-admission-mutate logsidecar-injector-admission-mutate mutating-webhook-configuration
dokubectl delete mutatingwebhookconfigurations.admissionregistration.k8s.io $webhook 2>/dev/null
done# delete users
for user in `kubectl get users -o jsonpath="{.items[*].metadata.name}"`
dokubectl patch user $user -p '{"metadata":{"finalizers":null}}' --type=merge
done
kubectl delete users --all 2>/dev/null# delete helm resources
for resource_type in `echo helmcategories helmapplications helmapplicationversions helmrepos helmreleases`; dofor resource_name in `kubectl get ${resource_type}.application.kubesphere.io -o jsonpath="{.items[*].metadata.name}"`; dokubectl patch ${resource_type}.application.kubesphere.io ${resource_name} -p '{"metadata":{"finalizers":null}}' --type=mergedonekubectl delete ${resource_type}.application.kubesphere.io --all 2>/dev/null
done# delete workspacetemplates
for workspacetemplate in `kubectl get workspacetemplates.tenant.kubesphere.io -o jsonpath="{.items[*].metadata.name}"`
dokubectl patch workspacetemplates.tenant.kubesphere.io $workspacetemplate -p '{"metadata":{"finalizers":null}}' --type=merge
done
kubectl delete workspacetemplates.tenant.kubesphere.io --all 2>/dev/null# delete federatednamespaces in namespace kubesphere-monitoring-federated
for resource in $(kubectl get federatednamespaces.types.kubefed.io -n kubesphere-monitoring-federated -oname); dokubectl patch "${resource}" -p '{"metadata":{"finalizers":null}}' --type=merge -n kubesphere-monitoring-federated
done# delete crds
for crd in `kubectl get crds -o jsonpath="{.items[*].metadata.name}"`
doif [[ $crd == *kubesphere.io ]] || [[ $crd == *kubefed.io ]] ; then kubectl delete crd $crd 2>/dev/null; fi
done# delete relevance ns
for ns in kube-federation-system kubesphere-alerting-system kubesphere-controls-system kubesphere-devops-system kubesphere-devops-worker kubesphere-logging-system kubesphere-monitoring-system kubesphere-monitoring-federated openpitrix-system kubesphere-system
dokubectl delete ns $ns 2>/dev/null
done

#执行删除
sh del.sh

十二、 GitLab安装

1. 单独准备服务器，采用Docker安装

docker search gitlab
docker pull gitlab/gitlab-ce

2.准备docker-compose.yml文件

mkdir -p /data/gitvim /data/git/docker-compose.ymlversion: '3.1'
services:gitlab:image: 'gitlab/gitlab-ce:latest'container_name: gitlabrestart: alwaysenvironment:GITLAB_OMNIBUS_CONFIG: |external_url 'http://10.1.100.225:8929'#自己安装git的服务器IPgitlab_rails['gitlab_shell_ssh_port'] = 2224ports:- '8929:8929'- '2224:2224'volumes:- './config:/etc/gitlab'- './logs:/var/log/gitlab'- './data:/var/opt/gitlab'

3.启动容器（需要稍等很久……）

cd /data/git
docker-compose up -d

4.访问GitLab首页

http://10.1.100.225:8929

5.查看root用户初始密码

docker exec -it gitlab cat /etc/gitlab/initial_root_password

6.第一次登录网页，需要修改密码 Password

**右上角>>**Administrator>Preferences>Password

十三、DevOps初始化环境

1.linux系统 安装Jenkins、jdk 、maven

1.下载地址

JDK包下载地址
https://www.oracle.com/java/technologies/downloads/

MAven下载地址
https://maven.apache.org/download.cgi

2.安装jdk maven

tar -zxvf jdk-8*.tar.gz -C /usr/local/
tar -zxvf apache-maven-*.tar.gz -C /usr/local/cd /usr/local
mv apache-maven*/ maven
mv jdk1.8*/ jdk

2.1 编辑maven配置

vim /usr/local/maven/conf/settings.xml

 

nexus-aliyuncentralNexus aliyunhttp://maven.aliyun.com/nexus/content/groups/public

    jdk1.8        true    1.8            1.8    1.8    1.8         
 jdk1.8

3.安装jenkins

3.1 下载

docker pull jenkins/jenkins:2.319.1-lts

3.2 创建yaml

mkdir -p /data/jenkins/
cd /data/jenkins/
vim /data/jenkins/docker-compose.yml

version: "3.1"
services:jenkins:image: jenkins/jenkinscontainer_name: jenkinsports:- 8080:8080- 50000:50000volumes:- ./data/:/var/jenkins_home/- /var/run/docker.sock:/var/run/docker.sock- /usr/bin/docker:/usr/bin/docker- /etc/docker/daemon.json:/etc/docker/daemon.json

3.3 启动jenkins

 
#修改Jenkins用户权限
cd /var/runchown root:root docker.sock#其他用户有读和写权限
chmod o+rw docker.sockcd /data/jenkins/
docker-compose up -d#授权
chmod 777 /data/jenkins/data/cat /data/jenkins/data/hudson.model.UpdateCenter.xml
#重新启动Jenkins容器后，由于Jenkins需要下载大量内容，但是由于默认下载地址下载速度较慢，
#需要重新设置下载地址为国内镜像站# 清华大学的插件源也可以
# 修改数据卷中的hudson.model.UpdateCenter.xml文件
# 将下载地址替换为http://mirror.esuni.jp/jenkins/updates/update-center.json# 清华大学的插件源也可以
#https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json#重启
docker-compose restart#查看日志
docker logs -f jenkins

3.4 访问页面，安装插件

http://10.1.100.225:8080
1.输入密码2.选择插件来安装3.点击安装

4.jenkins插件安装

中文界面>系统管理>插件管理>可选插件>搜索插件
英文界面> Manage Jenkins–Manage Plugins-Available>搜索插件
Locale
Localization
Git Parameter
Publish Over SSH

5. 配置jenkins

mv /usr/local/maven/ /data/jenkins/data/
mv /usr/local/jdk/ /data/jenkins/data/

5.1 加载本地jdk

Dashboard>系统管理>全局工具配置>Add JDK>去掉对钩 （√）自动安装
NAME

jdk8

JAVA_HOME

/var/jenkins_home/jdk/

5.1 加载本地maven

Dashboard>系统管理>全局工具配置>Add Maven>去掉对钩 （√）自动安装
NAME

maven

JAVA_HOME

/var/jenkins_home/maven/

Save Apply
保存 应用

运行mvn测试
mvn help:system

3.jenkins拉取测试

系统管理>系统配置>Publish over SSH>SSH Servers>Add

#自定义项目名称
name

test

#主机IP
Hostname

10.1.100.25

#主机用户名
Username

root

#拉取项目路径
Remote Directory

/data/work/mytest

点击高级
√ Use password authentication, or use a different key

#输入服务器密码
Passphrase / Password

xxxx

#点击 测试

Test ConfigurationSave Apply
保存 应用

4.Jenkins服务器设置免密登入k8s-mast服务器

#Jenkins服务器-进入jenkins容器
docker exec -it jenkins bash

#进入jenkins容器-生成免密登录公私钥,根据提示按回车
ssh-keygen -t rsa

#进入jenkins容器-查看jenkins 秘钥
cat /var/jenkins_home/.ssh/id_rsa.pub

#k8s-mast服务器中authorized_keys 加入Jenkins服务器秘钥
echo “xxxxxx” >> /root/.ssh/authorized_keys

十四、开发环境部署IDEA

工具下载:

链接：https://pan.baidu.com/s/1Jkyh_kgrT2o388Xiujbdeg?pwd=b7rx
提取码：b7rx

1. windows配置maven 和jdk

https://blog.csdn.net/weixin_46565024/article/details/122758111

2. IDEA简单得项目创建

File>New>ProjectSpring Initializr>NextType(选择Maven)>Java Version (选择8) > NextWeb> 勾选√Spring Web> Next>Finish