0%

Redis-Cache-Breakdown

关于Redis场景下简单的处理方式。

源码地址

缓存穿透和缓存击穿

  • 缓存穿透是指用户查询数据,在数据库没有,自然在缓存中也不会有。这样就导致用户查询的时候,在缓存中找不到,每次都要去数据库再查询一遍,然后返回空(相当于进行了两次无用的查询)。这样请求就绕过缓存直接查数据库,这也是经常提的缓存命中率问题。
  • 解决办法
    • 缓存空对象:代码维护较简单,但是效果不好。不过部分场景能使用。

演示环境

  • spring boot 2.4.5
  • mysql
  • Redis
  • 测试环境采用docker-compose

缓存穿透

  • 理想的情况下我们数据缓存到redis里面,客户访问数据的时候直接去缓存查询,如果没有缓存就去数据库查询。
  • 仔细想想这里其实有一个bug,必须是缓存有的数据才会去数据库查询,如果一直都是在查询一个没有得数据咋办呢,缓存会一直落空,而且每次都会去数据库查询。

缓存击穿

这个场景更多是应用在多线程的环境,这里衔接上面的代码,当并发量上来的时候,大量的查询集中查询某一个key,若这个时候key刚好失效了,就会导致请求全部进入数据中。这个就是所谓的缓存击穿。

常用解决思路

这里注意列举常用方案,根据实际情况使用

空值缓存

在访问缓存key之前,设置另一个短期key来锁住当前key的访问。这个方案可以降低持续缓存落空的情况。至少数据库被穿透的概率小很多了。

  • Redis方案
    • 如上图所示,当缓存和数据库中都没找到的时候就将查询条件作为key并在缓存中插入一个空的数据,并指定好过期时间。这样下次的请求就会被缓存命中。

代码实现

项目结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
.
├── docker-compose.yml # docker-compose 配置文件
├── docker-config
   ├── mysql
      ├── init
         └── 1_init.sql # 初始化测试库
      └── my.cnf
   ├── pwd.txt
   └── redis
   └── redis.conf
├── pom.xml
└── src
├── main
   ├── java
      └── cn
      └── z201
      └── redis
      ├── AppApplication.java
      ├── AppApplicationController.java
      ├── AppApplicationServiceImpl.java
      └── RedisConfig.java
   └── resources
   ├── application-dev.yml
   ├── application-test.yml
   ├── application.yml
   └── logback.xml
└── test
└── java
└── cn
└── z201
└── redis
└── AppApplicationTest.java # 单元测试

Docker-compose配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
version : '3'

networks:
network-redis-cache-breakdown:
driver: bridge

services:
web:
container_name: cn.z201.docker-redis-cache-breakdown
build:
context: .
dockerfile: .
image: cn.z201.docker-redis-cache-breakdown
networks:
- network-redis-cache-breakdown
expose:
- '9001'
ports:
- '9001:9001'
depends_on: # 等待其它服务启动完成
- mysql
- redis
links:
- mysql
- redis
healthcheck:
test: "/bin/netstat -anpt|grep 9001"
interval: 30s
timeout: 3s
retries: 1
mysql:
image: mysql:5.7
container_name: mysql5.7-dev-5
networks:
- network-redis-cache-breakdown
expose:
- '3306'
ports:
- '3306:3306'
volumes:
- ./docker-config/mysql/my.cnf:/etc/mysql/my.cnf # 映射数据库配置文件
- ./docker-config/mysql/init:/docker-entrypoint-initdb.d # 初始化数据库
command: [
'--character-set-server=utf8mb4',
'--collation-server=utf8mb4_unicode_ci',
'--lower_case_table_names=1',
'--default-time-zone=+8:00']
environment:
- MYSQL_ROOT_PASSWORD=root # 设置root密码
healthcheck:
test: "/bin/netstat -anpt|grep 3306"
interval: 30s
timeout: 3s
retries: 1
redis:
image: redis:5.0.5
container_name: redis5.0.6-dev-5
networks:
- network-redis-cache-breakdown
expose:
- '6379'
ports:
- '6379:6379'
volumes:
- ./docker-config/redis/redis.conf:/etc/redis.conf # 映射数据库配置文件
command: redis-server /etc/redis.conf # 启动redis命令
healthcheck:
test: "/bin/netstat -anpt|grep 6379"
interval: 30s
timeout: 3s
retries: 1

工具类

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
package cn.z201.redis;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;

import java.time.Duration;
import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;

/**
* @author z201.coding@gmail.com
**/
@Service
@Slf4j
public class AppApplicationServiceImpl {

public static final String INFO = "INFO:";

private static String LOCK_PREFIX = "LOCK:P";

@Autowired
private JdbcTemplate jdbcTemplate;

@Autowired
private RedisTemplate redisTemplate;

/**
* 根据用户组件获取数据
*
* @param id
* @return
*/
public Map<String, Object> findById(String id) {
String sql = "SELECT * FROM person WHERE id = ?";
Map<String, Object> person = jdbcTemplate.queryForMap(sql, id);
return person;
}

// 获取全部数据
public List<Map<String, Object>> all() {
List<Map<String, Object>> personList =
jdbcTemplate.queryForList("SELECT * FROM person");
return personList;
}

// redis set ex px
private boolean lock(String key, long timeout) {
Boolean result = redisTemplate.opsForValue().setIfAbsent(LOCK_PREFIX, key, Duration.ofSeconds(timeout));
if (result) {
log.info("lock");
}
return result;
}

// redis单节点解锁
private boolean unlock(String key) {
String script = "if redis.call('get',KEYS[1]) == ARGV[1]"
+ "then"
+ " return redis.call('del',KEYS[1])"
+ "else "
+ " return 0 "
+ "end";
String[] args = new String[]{key};
DefaultRedisScript<Long> redisScript = new DefaultRedisScript<>(script, Long.class);
Object result = redisTemplate.execute(redisScript, Collections.singletonList(LOCK_PREFIX), args);
if (Objects.equals(result, 1L)) {
log.info("unlock ok");
return true;
}
return false;
}

// 空值响应
public Map<String, Object> findCacheById(String id, Long lockOutTime, Long emptyOutTime) {
Map<String, Object> result = new HashMap<>();
Gson gson = new GsonBuilder().create();
String key = INFO + id;
Object data = redisTemplate.opsForValue().get(key);
if (data == null) {
try {
if (lock(id, lockOutTime)) {
log.info("init data ");
result = findById(id);
if (CollectionUtils.isEmpty(result)) {
// 插入一个空格进去
log.info("set empty ");
result = new HashMap<>();
redisTemplate.opsForValue().set(key, gson.toJson(result), emptyOutTime, TimeUnit.MILLISECONDS);
} else {
log.info("set data ");
redisTemplate.opsForValue().set(key, gson.toJson(result));
}
}
} finally {
unlock(id);
}
} else {
result = gson.fromJson(data.toString(), Map.class);
if (!CollectionUtils.isEmpty(result)) {
log.info("get cache");
}
}
return result;
}
}

测试入口

单元测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
package cn.z201.redis;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.reflect.TypeToken;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.data.redis.connection.RedisStringCommands;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.data.redis.core.types.Expiration;
import org.springframework.data.redis.serializer.RedisSerializer;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.springframework.util.CollectionUtils;

import java.lang.reflect.Type;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;

@Slf4j
@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = AppApplication.class, webEnvironment = SpringBootTest.WebEnvironment.NONE)
@AutoConfigureMockMvc
// 指定单元测试方法顺序
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class AppApplicationTest {

@Autowired
private RedisTemplate redisTemplate;

@Autowired
AppApplicationServiceImpl appApplicationService;

@Test
@Disabled
public void dataView() {
List<Map<String, Object>> personList =
appApplicationService.all();
Gson gson = new GsonBuilder().setPrettyPrinting().create();
log.info("personList \n {}", gson.toJson(personList));
Map<String, Object> objectMap = appApplicationService.findById(personList.get(0).get("id").toString());
log.info("person \n {}", gson.toJson(objectMap));
}

private void clean() {
Set<String> keys =
redisTemplate.keys("*");
redisTemplate.delete(keys);
}

private void testCacheBreakdownInfo() {
int count = 10;
String id = "3";
CountDownLatch countDownLatch = new CountDownLatch(count);
ExecutorService executorService = Executors.newFixedThreadPool(count);
for (int i = 0; i < count; i++) {
executorService.execute(() -> {
log.info(" user {} ", appApplicationService.findCacheById(id, 1000L, 10000L));
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
countDownLatch.countDown();
});
}
try {
countDownLatch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
executorService.shutdown();
}

@Test
@Disabled
public void test() throws InterruptedException {
clean();
for (int i = 0; i < 2; i++) {
long startTimes = System.currentTimeMillis();
testCacheBreakdownInfo();
long endTimes = System.currentTimeMillis();
long runTime = (endTimes - startTimes);
log.info("执行完毕: {}ms", runTime);
Thread.sleep(1000L);
}
clean();
}

}
  • Console
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[pool-2-thread-3]  lock
[pool-2-thread-3] init data
[pool-2-thread-3] HikariPool-1 - Starting...
[pool-2-thread-8] unlock ok
[pool-2-thread-4] user {}
[pool-2-thread-5] user {}
[pool-2-thread-9] user {}
[pool-2-thread-6] user {}
[pool-2-thread-7] user {}
[pool-2-thread-10] user {}
[pool-2-thread-8] user {}
[pool-2-thread-2] user {}
[pool-2-thread-1] user {}
[pool-2-thread-3] HikariPool-1 - Start completed.
[pool-2-thread-3] set data
[pool-2-thread-3] user {id=3, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=13100077310, sex=true, birth_date=1998-03-02 20:23:08.0, identity_card_number=460501198803127323}
[main] 执行完毕: 1336ms
[pool-3-thread-7] get cache
[pool-3-thread-10] get cache
[pool-3-thread-9] get cache
[pool-3-thread-4] get cache
[pool-3-thread-6] get cache
[pool-3-thread-5] get cache
[pool-3-thread-2] get cache
[pool-3-thread-3] get cache
[pool-3-thread-1] get cache
[pool-3-thread-8] get cache
[pool-3-thread-3] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-10] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-7] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-1] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-8] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-9] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-5] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-2] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-4] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-6] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[main] 执行完毕: 1025ms

接口测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package cn.z201.redis;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;

/**
* @author z201.coding@gmail.com
**/
@RestController
public class AppApplicationController {

@Autowired
AppApplicationServiceImpl appApplicationService;

@RequestMapping(value = "{id}")
public Object index(@PathVariable(required = false) String id) {
Map<String, Object> data = new HashMap<>();
data.put("code", "200");
if (null == id) {
data.put("data", "请求参数不合法");
}else{
data.put("data", appApplicationService.findCacheById(id,1000L,10000L));
}
return data;
}
}
  • Console
1
2
  docker-run curl http://127.0.0.1:9001/redis/1
{"code":"200","data":{"id":1,"full_name":"灵芸黄","email":"lingyun.huang@gmail.com","telephone_number":13842681484,"sex":true,"birth_date":"1996-05-23T21:17:45.000+00:00","identity_card_number":361405197211160626}}%

END