胖胖的枫叶

流水要争先，争的是滔滔不绝。

0%

Docker-Compose-Redis-Cache-Breakdown

发表于 更新于 分类于
本文字数： 17k 阅读时长 ≈ 15 分钟

关于Redis场景下简单的处理方式。

源码地址

缓存穿透和缓存击穿

  • 缓存穿透是指用户查询数据，在数据库没有，自然在缓存中也不会有。这样就导致用户查询的时候，在缓存中找不到，每次都要去数据库再查询一遍，然后返回空（相当于进行了两次无用的查询）。这样请求就绕过缓存直接查数据库，这也是经常提的缓存命中率问题。
  • 解决办法
    • 缓存空对象：代码维护较简单，但是效果不好。不过部分场景能使用。

演示环境

  • spring boot 2.4.5
  • mysql
  • Redis
  • 测试环境采用docker-compose

缓存穿透

  • 理想的情况下我们数据缓存到redis里面，客户访问数据的时候直接去缓存查询，如果没有缓存就去数据库查询。
  • 仔细想想这里其实有一个bug，必须是缓存有的数据才会去数据库查询，如果一直都是在查询一个没有得数据咋办呢，缓存会一直落空，而且每次都会去数据库查询。

缓存击穿

这个场景更多是应用在多线程的环境，这里衔接上面的代码，当并发量上来的时候，大量的查询集中查询某一个key，若这个时候key刚好失效了，就会导致请求全部进入数据中。这个就是所谓的缓存击穿。

常用解决思路

这里注意列举常用方案，根据实际情况使用

空值缓存

在访问缓存key之前，设置另一个短期key来锁住当前key的访问。这个方案可以降低持续缓存落空的情况。至少数据库被穿透的概率小很多了。

  • Redis方案
    • 如上图所示，当缓存和数据库中都没找到的时候就将查询条件作为key并在缓存中插入一个空的数据，并指定好过期时间。这样下次的请求就会被缓存命中。

代码实现

项目结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
.
├── docker-compose.yml # docker-compose 配置文件
├── docker-config
│   ├── mysql
│   │   ├── init
│   │   │   └── 1_init.sql # 初始化测试库
│   │   └── my.cnf
│   ├── pwd.txt
│   └── redis
│       └── redis.conf
├── pom.xml
└── src
    ├── main
    │   ├── java
    │   │   └── cn
    │   │       └── z201
    │   │           └── redis
    │   │               ├── AppApplication.java
    │   │               ├── AppApplicationController.java
    │   │               ├── AppApplicationServiceImpl.java
    │   │               └── RedisConfig.java
    │   └── resources
    │       ├── application-dev.yml
    │       ├── application-test.yml
    │       ├── application.yml
    │       └── logback.xml
    └── test
        └── java
            └── cn
                └── z201
                    └── redis
                        └── AppApplicationTest.java # 单元测试


Docker-compose配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
version : '3'

networks:
  network-redis-cache-breakdown:
    driver: bridge

services:
  web:
    container_name: cn.z201.docker-redis-cache-breakdown
    build:
      context: .
      dockerfile: .
    image: cn.z201.docker-redis-cache-breakdown
    networks:
      - network-redis-cache-breakdown
    expose:
      - '9001'
    ports:
      - '9001:9001'
    depends_on: # 等待其它服务启动完成
      - mysql
      - redis
    links:
      - mysql
      - redis
    healthcheck:
      test: "/bin/netstat -anpt|grep 9001"
      interval: 30s
      timeout: 3s
      retries: 1
  mysql:
    image: mysql:5.7
    container_name: mysql5.7-dev-5
    networks:
      - network-redis-cache-breakdown
    expose:
      - '3306'
    ports:
      - '3306:3306'
    volumes:
      - ./docker-config/mysql/my.cnf:/etc/mysql/my.cnf # 映射数据库配置文件
      - ./docker-config/mysql/init:/docker-entrypoint-initdb.d # 初始化数据库
    command: [
      '--character-set-server=utf8mb4',
      '--collation-server=utf8mb4_unicode_ci',
      '--lower_case_table_names=1',
      '--default-time-zone=+8:00']
    environment:
      - MYSQL_ROOT_PASSWORD=root # 设置root密码
    healthcheck:
      test: "/bin/netstat -anpt|grep 3306"
      interval: 30s
      timeout: 3s
      retries: 1
  redis:
    image: redis:5.0.5
    container_name: redis5.0.6-dev-5
    networks:
      - network-redis-cache-breakdown
    expose:
      - '6379'
    ports:
      - '6379:6379'
    volumes:
      - ./docker-config/redis/redis.conf:/etc/redis.conf # 映射数据库配置文件
    command: redis-server /etc/redis.conf # 启动redis命令
    healthcheck:
      test: "/bin/netstat -anpt|grep 6379"
      interval: 30s
      timeout: 3s
      retries: 1

工具类

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
package cn.z201.redis;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;

import java.time.Duration;
import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;

/**
 * @author z201.coding@gmail.com
 **/
@Service
@Slf4j
public class AppApplicationServiceImpl {

    public static final String INFO = "INFO:";

    private static String LOCK_PREFIX = "LOCK:P";

    @Autowired
    private JdbcTemplate jdbcTemplate;

    @Autowired
    private RedisTemplate redisTemplate;

    /**
     * 根据用户组件获取数据
     *
     * @param id
     * @return
     */
    public Map<String, Object> findById(String id) {
        String sql = "SELECT * FROM person WHERE id = ?";
        Map<String, Object> person = jdbcTemplate.queryForMap(sql, id);
        return person;
    }
	
		// 获取全部数据  
    public List<Map<String, Object>> all() {
        List<Map<String, Object>> personList =
                jdbcTemplate.queryForList("SELECT * FROM person");
        return personList;
    }

  // redis set ex px 
    private boolean lock(String key, long timeout) {
        Boolean result = redisTemplate.opsForValue().setIfAbsent(LOCK_PREFIX, key, Duration.ofSeconds(timeout));
        if (result) {
            log.info("lock");
        }
        return result;
    }

  // redis单节点解锁
    private boolean unlock(String key) {
        String script = "if redis.call('get',KEYS[1]) == ARGV[1]"
                + "then"
                + "     return redis.call('del',KEYS[1])"
                + "else "
                + "     return 0 "
                + "end";
        String[] args = new String[]{key};
        DefaultRedisScript<Long> redisScript = new DefaultRedisScript<>(script, Long.class);
        Object result = redisTemplate.execute(redisScript, Collections.singletonList(LOCK_PREFIX), args);
        if (Objects.equals(result, 1L)) {
            log.info("unlock ok");
            return true;
        }
        return false;
    }

  // 空值响应
     public Map<String, Object> findCacheById(String id, Long lockOutTime, Long emptyOutTime) {
        Map<String, Object> result = new HashMap<>();
        Gson gson = new GsonBuilder().create();
        String key = INFO + id;
        Object data = redisTemplate.opsForValue().get(key);
        if (data == null) {
            try {
                if (lock(id, lockOutTime)) {
                    log.info("init data ");
                    result = findById(id);
                    if (CollectionUtils.isEmpty(result)) {
                        // 插入一个空格进去
                        log.info("set empty ");
                        result = new HashMap<>();
                        redisTemplate.opsForValue().set(key, gson.toJson(result), emptyOutTime, TimeUnit.MILLISECONDS);
                    } else {
                        log.info("set data ");
                        redisTemplate.opsForValue().set(key, gson.toJson(result));
                    }
                }
            } finally {
                unlock(id);
            }
        } else {
            result = gson.fromJson(data.toString(), Map.class);
            if (!CollectionUtils.isEmpty(result)) {
                log.info("get cache");
            }
        }
        return result;
    }
}

测试入口

单元测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
package cn.z201.redis;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.reflect.TypeToken;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.data.redis.connection.RedisStringCommands;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.data.redis.core.types.Expiration;
import org.springframework.data.redis.serializer.RedisSerializer;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.springframework.util.CollectionUtils;

import java.lang.reflect.Type;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;

@Slf4j
@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = AppApplication.class, webEnvironment = SpringBootTest.WebEnvironment.NONE)
@AutoConfigureMockMvc
// 指定单元测试方法顺序
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class AppApplicationTest {

    @Autowired
    private RedisTemplate redisTemplate;

    @Autowired
    AppApplicationServiceImpl appApplicationService;

    @Test
    @Disabled
    public void dataView() {
        List<Map<String, Object>> personList =
                appApplicationService.all();
        Gson gson = new GsonBuilder().setPrettyPrinting().create();
        log.info("personList \n {}", gson.toJson(personList));
        Map<String, Object> objectMap = appApplicationService.findById(personList.get(0).get("id").toString());
        log.info("person \n {}", gson.toJson(objectMap));
    }

    private void clean() {
        Set<String> keys =
                redisTemplate.keys("*");
        redisTemplate.delete(keys);
    }

    private void testCacheBreakdownInfo() {
        int count = 10;
        String id = "3";
        CountDownLatch countDownLatch = new CountDownLatch(count);
        ExecutorService executorService = Executors.newFixedThreadPool(count);
        for (int i = 0; i < count; i++) {
            executorService.execute(() -> {
                log.info(" user {} ", appApplicationService.findCacheById(id, 1000L, 10000L));
                try {
                    Thread.sleep(1000L);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                countDownLatch.countDown();
            });
        }
        try {
            countDownLatch.await();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        executorService.shutdown();
    }

    @Test
    @Disabled
    public void test() throws InterruptedException {
        clean();
        for (int i = 0; i < 2; i++) {
            long startTimes = System.currentTimeMillis();
            testCacheBreakdownInfo();
            long endTimes = System.currentTimeMillis();
            long runTime = (endTimes - startTimes);
            log.info("执行完毕:  {}ms", runTime);
            Thread.sleep(1000L);
        }
        clean();
    }

}
  • Console
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
[pool-2-thread-3]  lock
[pool-2-thread-3]  init data 
[pool-2-thread-3]  HikariPool-1 - Starting...
[pool-2-thread-8]  unlock ok
[pool-2-thread-4]   user {} 
[pool-2-thread-5]   user {} 
[pool-2-thread-9]   user {} 
[pool-2-thread-6]   user {} 
[pool-2-thread-7]   user {} 
[pool-2-thread-10]   user {} 
[pool-2-thread-8]   user {} 
[pool-2-thread-2]   user {} 
[pool-2-thread-1]   user {} 
[pool-2-thread-3]  HikariPool-1 - Start completed.
[pool-2-thread-3]  set data 
[pool-2-thread-3]   user {id=3, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=13100077310, sex=true, birth_date=1998-03-02 20:23:08.0, identity_card_number=460501198803127323} 
[main]  执行完毕:  1336ms
[pool-3-thread-7]  get cache
[pool-3-thread-10]  get cache
[pool-3-thread-9]  get cache
[pool-3-thread-4]  get cache
[pool-3-thread-6]  get cache
[pool-3-thread-5]  get cache
[pool-3-thread-2]  get cache
[pool-3-thread-3]  get cache
[pool-3-thread-1]  get cache
[pool-3-thread-8]  get cache
[pool-3-thread-3]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[pool-3-thread-10]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[pool-3-thread-7]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[pool-3-thread-1]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[pool-3-thread-8]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[pool-3-thread-9]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[pool-3-thread-5]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[pool-3-thread-2]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[pool-3-thread-4]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[pool-3-thread-6]   user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17} 
[main]  执行完毕:  1025ms

接口测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
package cn.z201.redis;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;

/**
 * @author z201.coding@gmail.com
 **/
@RestController
public class AppApplicationController {

    @Autowired
    AppApplicationServiceImpl appApplicationService;

    @RequestMapping(value = "{id}")
    public Object index(@PathVariable(required = false) String id) {
        Map<String, Object> data = new HashMap<>();
        data.put("code", "200");
        if (null == id) {
            data.put("data", "请求参数不合法");
        }else{
            data.put("data", appApplicationService.findCacheById(id,1000L,10000L));
        }
        return data;
    }
}

  • Console
1
2
  docker-run curl http://127.0.0.1:9001/redis/1
{"code":"200","data":{"id":1,"full_name":"灵芸黄","email":"lingyun.huang@gmail.com","telephone_number":13842681484,"sex":true,"birth_date":"1996-05-23T21:17:45.000+00:00","identity_card_number":361405197211160626}}%

END