Spring Cloud Kubernetes之实战一配置管理

2020-07-10

字数统计: 3.2k | 阅读时长约 17 分钟

一直以来，玩springcloud的，基本都是在玩Springboot1.x，Springcloud(Dalston版)的众多相关组件来做配置中心、服务注册与发现，网关用的是Netflix公司对springboot做的LB，等等，但是这些东西太过沉重，复杂了。在一个以k8s为基础的iaas服务系统，如果不用k8s的特性来做这些事，那是说不过去的。理由这就不重复述说了。一句话：减少系统服务的复杂性。

本文主要介绍springcloud结合k8s，做配置管理，避免更多服务组件的冗余，完美填坑版！

环境：

ubuntu16.04

docker18.04

k8s1.13.x +

maven3.5.3

java1.8 +

springboot 2.1.1

spring-cloud-kubernetes：1.0.1.RELEAS

前提

Ubuntu下安装docker18.04 or 其它较高版本，k8s1.13.x及以上，jvm环境等。

创建项目

基础依赖：

<parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.8.RELEASE</version>
        <relativePath/>
    </parent>
	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>1.8</java.version>
		<swagger.version>2.6.1</swagger.version>
		<xstream.version>1.4.7</xstream.version>
		<pageHelper.version>4.1.6</pageHelper.version>
		<fastjson.version>1.2.51</fastjson.version>
		<shiro.version>1.3.0</shiro.version>
		<!-- <kubernetes-client-version>6.0.1</kubernetes-client-version> -->
		<kubernetes-client-version>5.0.0</kubernetes-client-version>
		<fabric8-kubernetes-client.version>4.6.1</fabric8-kubernetes-client.version><!-- 对应k8s v1.15.3 -->
		<springcloud.version>Greenwich.SR4</springcloud.version>
		<springcloud.kubernetes.version>1.1.1.RELEASE</springcloud.kubernetes.version>
		<mysql.version>5.1.46</mysql.version>
	</properties>

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-dependencies</artifactId>
				<version>${springcloud.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

核心依赖：

<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
			<exclusions>
				<exclusion>
					<groupId>org.springframework.boot</groupId>
					<artifactId>spring-boot-starter-tomcat</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-undertow</artifactId>
		</dependency>

		<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-actuator</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-actuator-autoconfigure</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-kubernetes-config</artifactId>
            </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

<!-- springcloud-k8s-discovery -->

		<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-commons</artifactId>
        </dependency>

		<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-kubernetes-core</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-kubernetes-discovery</artifactId>
        </dependency>

		<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-kubernetes-ribbon</artifactId>
        </dependency>

		<dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
        </dependency>

    **本次依赖引入配置管理、服务的发现(即消费者)。**

如果有操作redis和db的话，引入相应的依赖：

<!-- mybatis -->
        <dependency>
            <groupId>org.mybatis.spring.boot</groupId>
            <artifactId>mybatis-spring-boot-starter</artifactId>
            <version>1.1.1</version>
        </dependency>
        <dependency>
			<groupId>mysql</groupId>
			<artifactId>mysql-connector-java</artifactId>
			<version>${mysql.version}</version>
		</dependency>

        <!--分页插件-->
	    <dependency>
	      <groupId>com.github.pagehelper</groupId>
	      <artifactId>pagehelper</artifactId>
	      <version>${pageHelper.version}</version>
	    </dependency>

        <!-- datasource pool-->
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>druid</artifactId>
            <version>1.1.3</version>
        </dependency>

		<dependency>
		    <groupId>org.apache.commons</groupId>
		    <artifactId>commons-lang3</artifactId>
		    </dependency>

		<dependency>
		    <groupId>commons-collections</groupId>
		    <artifactId>commons-collections</artifactId>
		    <version>3.2.2</version>
		</dependency>

        <!-- 对redis支持,引入的话项目缓存就支持redis了,所以必须加上redis的相关配置,否则操作相关缓存会报异常 -->
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-data-redis</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-redis</artifactId>
			<version>1.4.7.RELEASE</version>
		</dependency>

		<dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>19.0</version>
        </dependency>

剩下的就是构建镜像时的插件：

<build>
		<finalName>${project.artifactId}</finalName>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
				<configuration>
					<jvmArguments>-Dfile.encoding=UTF-8</jvmArguments>
					<fork>true</fork>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.jacoco</groupId>
				<artifactId>jacoco-maven-plugin</artifactId>
				<version>0.7.8</version>
				<executions>
					<execution>
						<goals>
							<goal>prepare-agent</goal>
							<goal>report</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>

接下来，我们创建主类：

@SpringBootApplication(scanBasePackages = { "com.leinao" })
@EnableConfigurationProperties(EnvConfig.class)
@EnableDiscoveryClient
@EnableHystrix
@EnableScheduling
public class AdminApp {
	public static void main(String[] args) {
		SpringApplication.run(AdminApp.class, args);
	}
}

注意这里创建启动类时，对springboot的项目进行了优化，避免启动时加载很多，启动繁重，具体深度优化，可参考：https://mp.weixin.qq.com/s?__biz=MzU2NjIzNDk5NQ==&mid=2247487954&idx=1&sn=2426451f3bd83161cfe1237f82d6b448&key=f8fb043b3d2681a794e51a46e142af77355722dff712776af12b1f3c831218df6dfc329df63c8e5e550b3d88d58f0f178c4c3c16b141733e0e3344fa595e2bc25241d864d45132753fd99279b832de85&ascene=1&uin=MzQzMzI2NjAxMQ%3D%3D&devicetype=Windows+10&version=62070158&lang=zh_CN&pass_ticket=pnSSI9jAq0M11V5hYMmkoVm5qO%2FWk9l3UUUJMglbdtdDOzLHa7iHsDmwSzs486sD。

然后我们在进行配置，注意：据官方说，项目的src\main\resources路径下不要创建application.yml文件，只创建名为bootstrap.yml的文件：

management:
  endpoint:
    restart:
      enabled: true
    health:
      enabled: true
    info:
      enabled: true

spring:
  application:
    name: edge-admin
  cloud:
    kubernetes:
      config:
        sources:
         - name: ${spring.application.name}
           namespace: default
      discovery:
        all-namespaces: true
      reload:
        #自动更新配置的开关设置为打开
        enabled: true
        #更新配置信息的模式：polling是主动拉取，event是事件通知
        mode: polling
        #主动拉取的间隔时间是500毫秒
        period: 500
  http:
    encoding:
      charset: UTF-8
      enabled: true
      force: true
  mvc:
    throw-exception-if-no-handler-found: true
  main:
    allow-bean-definition-overriding: true # 当遇到同样名称时，是否允许覆盖注册

这里，我创建了bootstrap文件，同时也加了application文件，启动时会先加载bootstrap，验证有效。

在application.yaml中，我们加入如下内容：

server:
  port: 9999
  undertow:
    accesslog:
      enabled: false
      pattern: combined
  servlet:
    session:
      timeout: PT120M

logging:
  path: /data/${spring.application.name}/logs

management:
  endpoint:
    restart:
      enabled: true
    health:
      enabled: true
    info:
      enabled: true
client:
  http:
    request:
      connectTimeout: 8000
      readTimeout: 30000

mybatis:
  mapperLocations: classpath:mapper/*.xml
  typeAliasesPackage: com.demo.*.model

backend:
  ribbon:
    eureka:
      enabled: false
    client:
      enabled: true
    ServerListRefreshInterval: 5000
hystrix.command.BackendCall.execution.isolation.thread.timeoutInMilliseconds: 5000
hystrix.threadpool.BackendCallThread.coreSize: 5

注意：这里的server设置session的超时时间，对于springboot2.0与1.0版本完全不一样了，具体看内容。

其他的application-test.yaml等配置文件，配置的是日志的级别：

logging:
  level:
    com:
      leinao: INFO
    org:
      springframework:
        web: INFO

接下来配置环境配置：

EnvConfig.java类作为环境变量配置，注解ConfigurationProperties的prefix=”spring_cloud”，

表示该类用到的配置项都是名为”spring_cloud”的配置项的子内容：

package com.demo.config;

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;

/**
 * 配置信息
 * @author Damon
 * @date 2019年10月25日 下午1:54:01
 *
 */

@Configuration
@ConfigurationProperties(prefix = "greeting")
public class EnvConfig {

    private String message = "This is a dummy message";

    private String container_command;
    private String model_dir_path;
    private String so_path;
    private String config_path;
    private String task_role_name;
    private String container_name;
    private String container_workdir;
    private String init_containers_image;
    private String service_account_name;
    private String spring_mq_host;
    private String spring_mq_port;
    private String spring_mq_user;
    private String spring_mq_pwd;
    private String jdbc_driverClassName;
    private String jdbc_url;
    private String jdbc_username;
    private String jdbc_password;
    private String spring_redis_host;
    private String spring_redis_port;
    private String spring_redis_pwd;
    private String kube_apiserver_address;
    private String image_path;
    private String volume_image_path;
    private String inference_job_namespace;
    private String api_version;
    private String remote_deployment_url;
    private String remote_pods_url;
    private String remote_deployment_pod_log_url;
    private String base_path;
    private String chunk_size;
    private String cas_url;
    private String create_job_url;
    private String abnormal_data_dir;

    private Long expire_time= 600000L;

    public String getMessage() {
        return this.message;
    }

    public void setMessage(String message) {
        this.message = message;
    }

	public String getContainer_command() {
		return container_command;
	}

	public void setContainer_command(String container_command) {
		this.container_command = container_command;
	}

	public String getModel_dir_path() {
		return model_dir_path;
	}

	public void setModel_dir_path(String model_dir_path) {
		this.model_dir_path = model_dir_path;
	}

	public String getSo_path() {
		return so_path;
	}

	public void setSo_path(String so_path) {
		this.so_path = so_path;
	}

	public String getConfig_path() {
		return config_path;
	}

	public void setConfig_path(String config_path) {
		this.config_path = config_path;
	}

	public String getTask_role_name() {
		return task_role_name;
	}

	public void setTask_role_name(String task_role_name) {
		this.task_role_name = task_role_name;
	}

	public String getContainer_name() {
		return container_name;
	}

	public void setContainer_name(String container_name) {
		this.container_name = container_name;
	}

	public String getContainer_workdir() {
		return container_workdir;
	}

	public void setContainer_workdir(String container_workdir) {
		this.container_workdir = container_workdir;
	}

	public String getInit_containers_image() {
		return init_containers_image;
	}

	public void setInit_containers_image(String init_containers_image) {
		this.init_containers_image = init_containers_image;
	}

	public String getService_account_name() {
		return service_account_name;
	}

	public void setService_account_name(String service_account_name) {
		this.service_account_name = service_account_name;
	}

	public String getSpring_mq_host() {
		return spring_mq_host;
	}

	public void setSpring_mq_host(String spring_mq_host) {
		this.spring_mq_host = spring_mq_host;
	}

	public String getSpring_mq_port() {
		return spring_mq_port;
	}

	public void setSpring_mq_port(String spring_mq_port) {
		this.spring_mq_port = spring_mq_port;
	}

	public String getSpring_mq_user() {
		return spring_mq_user;
	}

	public void setSpring_mq_user(String spring_mq_user) {
		this.spring_mq_user = spring_mq_user;
	}

	public String getSpring_mq_pwd() {
		return spring_mq_pwd;
	}

	public void setSpring_mq_pwd(String spring_mq_pwd) {
		this.spring_mq_pwd = spring_mq_pwd;
	}

	public String getJdbc_driverClassName() {
		return jdbc_driverClassName;
	}

	public void setJdbc_driverClassName(String jdbc_driverClassName) {
		this.jdbc_driverClassName = jdbc_driverClassName;
	}

	public String getJdbc_url() {
		return jdbc_url;
	}

	public void setJdbc_url(String jdbc_url) {
		this.jdbc_url = jdbc_url;
	}

	public String getJdbc_username() {
		return jdbc_username;
	}

	public void setJdbc_username(String jdbc_username) {
		this.jdbc_username = jdbc_username;
	}

	public String getJdbc_password() {
		return jdbc_password;
	}

	public void setJdbc_password(String jdbc_password) {
		this.jdbc_password = jdbc_password;
	}

	public String getSpring_redis_host() {
		return spring_redis_host;
	}

	public void setSpring_redis_host(String spring_redis_host) {
		this.spring_redis_host = spring_redis_host;
	}

	public String getSpring_redis_port() {
		return spring_redis_port;
	}

	public void setSpring_redis_port(String spring_redis_port) {
		this.spring_redis_port = spring_redis_port;
	}

	public String getSpring_redis_pwd() {
		return spring_redis_pwd;
	}

	public void setSpring_redis_pwd(String spring_redis_pwd) {
		this.spring_redis_pwd = spring_redis_pwd;
	}

	public String getKube_apiserver_address() {
		return kube_apiserver_address;
	}

	public void setKube_apiserver_address(String kube_apiserver_address) {
		this.kube_apiserver_address = kube_apiserver_address;
	}

	public String getImage_path() {
		return image_path;
	}

	public void setImage_path(String image_path) {
		this.image_path = image_path;
	}

	public String getVolume_image_path() {
		return volume_image_path;
	}

	public void setVolume_image_path(String volume_image_path) {
		this.volume_image_path = volume_image_path;
	}

	public String getInference_job_namespace() {
		return inference_job_namespace;
	}

	public void setInference_job_namespace(String inference_job_namespace) {
		this.inference_job_namespace = inference_job_namespace;
	}

	public String getApi_version() {
		return api_version;
	}

	public void setApi_version(String api_version) {
		this.api_version = api_version;
	}

	public String getRemote_deployment_url() {
		return remote_deployment_url;
	}

	public void setRemote_deployment_url(String remote_deployment_url) {
		this.remote_deployment_url = remote_deployment_url;
	}

	public String getRemote_pods_url() {
		return remote_pods_url;
	}

	public void setRemote_pods_url(String remote_pods_url) {
		this.remote_pods_url = remote_pods_url;
	}

	public String getRemote_deployment_pod_log_url() {
		return remote_deployment_pod_log_url;
	}

	public void setRemote_deployment_pod_log_url(String remote_deployment_pod_log_url) {
		this.remote_deployment_pod_log_url = remote_deployment_pod_log_url;
	}

	public String getBase_path() {
		return base_path;
	}

	public void setBase_path(String base_path) {
		this.base_path = base_path;
	}

	public String getChunk_size() {
		return chunk_size;
	}

	public void setChunk_size(String chunk_size) {
		this.chunk_size = chunk_size;
	}

	public Long getExpire_time() {
		return expire_time;
	}

	public void setExpire_time(Long expire_time) {
		this.expire_time = expire_time;
	}

	public String getCas_url() {
		return cas_url;
	}

	public void setCas_url(String cas_url) {
		this.cas_url = cas_url;
	}

	public String getCreate_job_url() {
		return create_job_url;
	}

	public void setCreate_job_url(String create_job_url) {
		this.create_job_url = create_job_url;
	}

	public String getAbnormal_data_dir() {
		return abnormal_data_dir;
	}

	public void setAbnormal_data_dir(String abnormal_data_dir) {
		this.abnormal_data_dir = abnormal_data_dir;
	}


}

测试demo类：

/**
 * @author Damon
 * @date 2019年12月27日 上午9:16:41
 *
 */

@RestController
public class DemoController {

	@Autowired
    private EnvConfig envConfig;

	/**
	 *
	 * @author Damon
	 * @date 2019年12月26日
	 *
	 */

	@GetMapping(value = "/getTest")
	public String getTest() {
		return envConfig.getBase_path();
	}
}

重点：默认的svc是没有权限访问k8s的API Server的资源的，执行如下脚本，可以提升权限，允许其访问configmap的可读权限：

#使用这个代表集群最高权限，deployment中无需引入serviceAccount: config-reader
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fabric8-rbac
subjects:
  - kind: ServiceAccount
    # Reference to upper's `metadata.name`
    name: default
    # Reference to upper's `metadata.namespace`
    namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

配置configmap：

kind: ConfigMap
apiVersion: v1
metadata:
  name: edge-admin
data:
  application.yaml: |-
    greeting:
      message: Say Hello to the World
    ---
    spring:
      profiles: dev
    greeting:
      message: Say Hello to the Developers
    ---
    spring:
      profiles: test
    greeting:
      message: Say Hello to the Test
    ---
    spring:
      profiles: prod
    greeting:
      message: Say Hello to the Prod

接下来就是执行deployment启动项目了：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: edge-admin-deployment
  labels:
    app: edge-admin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: edge-admin
  template:
    metadata:
      labels:
        app: edge-admin
    spec:
      nodeSelector:
        edge-admin: "true"
      containers:
      - name: edge-admin
        image: 10.11.2.20:8000/harbor/edge-admin
        imagePullPolicy: IfNotPresent
        ports:
          - name: admin01
            containerPort: 1002
        volumeMounts:
        - mountPath: /home/edge-admin
          name: edge-admin-path
        - mountPath: /data/edge-admin
          name: edge-admin-log-path
        - mountPath: /etc/kubernetes
          name: kube-config-path
        - mountPath: /abnormal_data_dir
          name: abnormal-data-dir
        args: ["sh", "-c", "nohup java $JAVA_OPTS -jar -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -Xms1024m -Xmx1024m -Xmn256m -Xss256k -XX:SurvivorRatio=8 -XX:+UseConcMarkSweepGC edge-admin.jar --spring.profiles.active=dev", "&"]
      hostAliases:
      - ip: "10.10.1.5"
        hostnames:
        - "k8s.api.server"
        - "foo.remote"
      - ip: "127.0.0.1"
        hostnames:
        - "foo.localhost"
      - ip: "0.0.0.0"
        hostnames:
        - "foo.all"
      #利用admin-rbac.yaml来获取权限
      #serviceAccount: config-reader
      #serviceAccountName: config-reader
      volumes:
      - name: edge-admin-path
        hostPath:
          path: /var/pai/edge-admin
      - name: edge-admin-log-path
        hostPath:
          path: /data/edge-admin
      - name: kube-config-path
        hostPath:
          path: /etc/kubernetes
      - name: abnormal-data-dir
        hostPath:
          path: /data/images/detect_result/defect

其中，前面说的，项目启动参数对其性能优化，是对jvm的参数设置。分别执行kubectl apply -f deployment.yaml和configmap.yaml，创建demo时所用的configmap的资源以及利用k8s部署启动项目。

最后打开浏览器：执行ip:port/hello，即可看到configmap中对应的属性值，这里就不展示了，有兴趣的可以试试。

以上即是对springcloud和k8s首次结合后利用其configmap特性，来做配置管理，摒弃springcloud-config、spring-boot-starter-actuator的组件，减少系统的复杂性，毕竟k8s是肯定会被用到的，所以可以直接用其特性来做系统服务的环境配置管理。

结束福利

开源实战利用 k8s 作微服务的架构设计代码：

1
2
3

https://gitee.com/damon_one/spring-cloud-k8s
https://gitee.com/damon_one/spring-cloud-oauth2
https://gitee.com/damon_one/Springcloud-Learning-Dalston

欢迎大家 star，多多指教。

关于作者

笔名：Damon，技术爱好者，长期从事 Java 开发、Spring Cloud 的微服务架构设计，以及结合 Docker、K8s 做微服务容器化，自动化部署等一站式项目部署、落地。目前主要从事基于 K8s 云原生架构研发的工作。Golang 语言开发，长期研究边缘计算框架 KubeEdge、调度框架 Volcano 等。公众号 交个朋友之猿天地 发起人。个人微信 DamonStatham，星球：《交个朋友之猿田地》，个人网站：交个朋友之猿天地 | 微服务 | 容器化 | 自动化，欢迎來撩。

欢迎关注：InfoQ

欢迎关注：腾讯自媒体专栏

欢迎关注

公号：交个朋友之猿天地

公号：damon8

公号：天山六路折梅手

打赏

版权声明： 本博客所有文章除特别声明外，著作权归作者所有。转载请注明出处！