关于Redis场景下简单的处理方式。
缓存穿透和缓存击穿
- 缓存穿透是指用户查询数据，在数据库没有，自然在缓存中也不会有。这样就导致用户查询的时候，在缓存中找不到，每次都要去数据库再查询一遍，然后返回空（相当于进行了两次无用的查询）。这样请求就绕过缓存直接查数据库，这也是经常提的缓存命中率问题。
- 解决办法
- 缓存空对象：代码维护较简单，但是效果不好。不过部分场景能使用。
演示环境
- spring boot 2.4.5
- mysql
- Redis
- 测试环境采用docker-compose
缓存穿透
- 理想的情况下我们数据缓存到redis里面，客户访问数据的时候直接去缓存查询，如果没有缓存就去数据库查询。
- 仔细想想这里其实有一个bug，必须是缓存有的数据才会去数据库查询，如果一直都是在查询一个没有得数据咋办呢，缓存会一直落空，而且每次都会去数据库查询。
缓存击穿
这个场景更多是应用在多线程的环境，这里衔接上面的代码，当并发量上来的时候，大量的查询集中查询某一个key，若这个时候key刚好失效了，就会导致请求全部进入数据中。这个就是所谓的缓存击穿。
常用解决思路
这里注意列举常用方案，根据实际情况使用
空值缓存
在访问缓存key之前，设置另一个短期key来锁住当前key的访问。这个方案可以降低持续缓存落空的情况。至少数据库被穿透的概率小很多了。
- Redis方案
- 如上图所示，当缓存和数据库中都没找到的时候就将查询条件作为key并在缓存中插入一个空的数据，并指定好过期时间。这样下次的请求就会被缓存命中。
代码实现
项目结构
.
├── docker-compose.yml # docker-compose 配置文件
├── docker-config
│ ├── mysql
│ │ ├── init
│ │ │ └── 1_init.sql # 初始化测试库
│ │ └── my.cnf
│ ├── pwd.txt
│ └── redis
│ └── redis.conf
├── pom.xml
└── src
├── main
│ ├── java
│ │ └── cn
│ │ └── z201
│ │ └── redis
│ │ ├── AppApplication.java
│ │ ├── AppApplicationController.java
│ │ ├── AppApplicationServiceImpl.java
│ │ └── RedisConfig.java
│ └── resources
│ ├── application-dev.yml
│ ├── application-test.yml
│ ├── application.yml
│ └── logback.xml
└── test
└── java
└── cn
└── z201
└── redis
└── AppApplicationTest.java # 单元测试
Docker-compose配置文件
version : '3'
networks:
network-redis-cache-breakdown:
driver: bridge
services:
web:
container_name: cn.z201.docker-redis-cache-breakdown
build:
context: .
dockerfile: .
image: cn.z201.docker-redis-cache-breakdown
networks:
- network-redis-cache-breakdown
expose:
- '9001'
ports:
- '9001:9001'
depends_on: # 等待其它服务启动完成
- mysql
- redis
links:
- mysql
- redis
healthcheck:
test: "/bin/netstat -anpt|grep 9001"
interval: 30s
timeout: 3s
retries: 1
mysql:
image: mysql:5.7
container_name: mysql5.7-dev-5
networks:
- network-redis-cache-breakdown
expose:
- '3306'
ports:
- '3306:3306'
volumes:
- ./docker-config/mysql/my.cnf:/etc/mysql/my.cnf # 映射数据库配置文件
- ./docker-config/mysql/init:/docker-entrypoint-initdb.d # 初始化数据库
command: [
'--character-set-server=utf8mb4',
'--collation-server=utf8mb4_unicode_ci',
'--lower_case_table_names=1',
'--default-time-zone=+8:00']
environment:
- MYSQL_ROOT_PASSWORD=root # 设置root密码
healthcheck:
test: "/bin/netstat -anpt|grep 3306"
interval: 30s
timeout: 3s
retries: 1
redis:
image: redis:5.0.5
container_name: redis5.0.6-dev-5
networks:
- network-redis-cache-breakdown
expose:
- '6379'
ports:
- '6379:6379'
volumes:
- ./docker-config/redis/redis.conf:/etc/redis.conf # 映射数据库配置文件
command: redis-server /etc/redis.conf # 启动redis命令
healthcheck:
test: "/bin/netstat -anpt|grep 6379"
interval: 30s
timeout: 3s
retries: 1
工具类
package cn.z201.redis;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;
/**
* @author z201.coding@gmail.com
**/
@Service
@Slf4j
public class AppApplicationServiceImpl {
public static final String INFO = "INFO:";
private static String LOCK_PREFIX = "LOCK:P";
@Autowired
private JdbcTemplate jdbcTemplate;
@Autowired
private RedisTemplate redisTemplate;
/**
* 根据用户组件获取数据
*
* @param id
* @return
*/
public Map<String, Object> findById(String id) {
String sql = "SELECT * FROM person WHERE id = ?";
Map<String, Object> person = jdbcTemplate.queryForMap(sql, id);
return person;
}
// 获取全部数据
public List<Map<String, Object>> all() {
List<Map<String, Object>> personList =
jdbcTemplate.queryForList("SELECT * FROM person");
return personList;
}
// redis set ex px
private boolean lock(String key, long timeout) {
Boolean result = redisTemplate.opsForValue().setIfAbsent(LOCK_PREFIX, key, Duration.ofSeconds(timeout));
if (result) {
log.info("lock");
}
return result;
}
// redis单节点解锁
private boolean unlock(String key) {
String script = "if redis.call('get',KEYS[1]) == ARGV[1]"
+ "then"
+ " return redis.call('del',KEYS[1])"
+ "else "
+ " return 0 "
+ "end";
String[] args = new String[]{key};
DefaultRedisScript<Long> redisScript = new DefaultRedisScript<>(script, Long.class);
Object result = redisTemplate.execute(redisScript, Collections.singletonList(LOCK_PREFIX), args);
if (Objects.equals(result, 1L)) {
log.info("unlock ok");
return true;
}
return false;
}
// 空值响应
public Map<String, Object> findCacheById(String id, Long lockOutTime, Long emptyOutTime) {
Map<String, Object> result = new HashMap<>();
Gson gson = new GsonBuilder().create();
String key = INFO + id;
Object data = redisTemplate.opsForValue().get(key);
if (data == null) {
try {
if (lock(id, lockOutTime)) {
log.info("init data ");
result = findById(id);
if (CollectionUtils.isEmpty(result)) {
// 插入一个空格进去
log.info("set empty ");
result = new HashMap<>();
redisTemplate.opsForValue().set(key, gson.toJson(result), emptyOutTime, TimeUnit.MILLISECONDS);
} else {
log.info("set data ");
redisTemplate.opsForValue().set(key, gson.toJson(result));
}
}
} finally {
unlock(id);
}
} else {
result = gson.fromJson(data.toString(), Map.class);
if (!CollectionUtils.isEmpty(result)) {
log.info("get cache");
}
}
return result;
}
}
测试入口
单元测试
package cn.z201.redis;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.reflect.TypeToken;
import lombok.extern.slf4j.Slf4j;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.AutoConfigureMockMvc;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.data.redis.connection.RedisStringCommands;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.data.redis.core.types.Expiration;
import org.springframework.data.redis.serializer.RedisSerializer;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.springframework.util.CollectionUtils;
import java.lang.reflect.Type;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.ReentrantLock;
@Slf4j
@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = AppApplication.class, webEnvironment = SpringBootTest.WebEnvironment.NONE)
@AutoConfigureMockMvc
// 指定单元测试方法顺序
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class AppApplicationTest {
@Autowired
private RedisTemplate redisTemplate;
@Autowired
AppApplicationServiceImpl appApplicationService;
@Test
@Disabled
public void dataView() {
List<Map<String, Object>> personList =
appApplicationService.all();
Gson gson = new GsonBuilder().setPrettyPrinting().create();
log.info("personList \n {}", gson.toJson(personList));
Map<String, Object> objectMap = appApplicationService.findById(personList.get(0).get("id").toString());
log.info("person \n {}", gson.toJson(objectMap));
}
private void clean() {
Set<String> keys =
redisTemplate.keys("*");
redisTemplate.delete(keys);
}
private void testCacheBreakdownInfo() {
int count = 10;
String id = "3";
CountDownLatch countDownLatch = new CountDownLatch(count);
ExecutorService executorService = Executors.newFixedThreadPool(count);
for (int i = 0; i < count; i++) {
executorService.execute(() -> {
log.info(" user {} ", appApplicationService.findCacheById(id, 1000L, 10000L));
try {
Thread.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
countDownLatch.countDown();
});
}
try {
countDownLatch.await();
} catch (InterruptedException e) {
e.printStackTrace();
}
executorService.shutdown();
}
@Test
@Disabled
public void test() throws InterruptedException {
clean();
for (int i = 0; i < 2; i++) {
long startTimes = System.currentTimeMillis();
testCacheBreakdownInfo();
long endTimes = System.currentTimeMillis();
long runTime = (endTimes - startTimes);
log.info("执行完毕: {}ms", runTime);
Thread.sleep(1000L);
}
clean();
}
}
- Console
[pool-2-thread-3] lock
[pool-2-thread-3] init data
[pool-2-thread-3] HikariPool-1 - Starting...
[pool-2-thread-8] unlock ok
[pool-2-thread-4] user {}
[pool-2-thread-5] user {}
[pool-2-thread-9] user {}
[pool-2-thread-6] user {}
[pool-2-thread-7] user {}
[pool-2-thread-10] user {}
[pool-2-thread-8] user {}
[pool-2-thread-2] user {}
[pool-2-thread-1] user {}
[pool-2-thread-3] HikariPool-1 - Start completed.
[pool-2-thread-3] set data
[pool-2-thread-3] user {id=3, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=13100077310, sex=true, birth_date=1998-03-02 20:23:08.0, identity_card_number=460501198803127323}
[main] 执行完毕: 1336ms
[pool-3-thread-7] get cache
[pool-3-thread-10] get cache
[pool-3-thread-9] get cache
[pool-3-thread-4] get cache
[pool-3-thread-6] get cache
[pool-3-thread-5] get cache
[pool-3-thread-2] get cache
[pool-3-thread-3] get cache
[pool-3-thread-1] get cache
[pool-3-thread-8] get cache
[pool-3-thread-3] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-10] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-7] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-1] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-8] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-9] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-5] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-2] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-4] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[pool-3-thread-6] user {id=3.0, full_name=黛玉赵, email=daiyu.zhao@gmail.com, telephone_number=1.310007731E10, sex=true, birth_date=Mar 2, 1998 8:23:08 PM, identity_card_number=4.605011988031273E17}
[main] 执行完毕: 1025ms
接口测试
package cn.z201.redis;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
/**
* @author z201.coding@gmail.com
**/
@RestController
public class AppApplicationController {
@Autowired
AppApplicationServiceImpl appApplicationService;
@RequestMapping(value = "{id}")
public Object index(@PathVariable(required = false) String id) {
Map<String, Object> data = new HashMap<>();
data.put("code", "200");
if (null == id) {
data.put("data", "请求参数不合法");
}else{
data.put("data", appApplicationService.findCacheById(id,1000L,10000L));
}
return data;
}
}
- Console
docker-run curl http://127.0.0.1:9001/redis/1
{"code":"200","data":{"id":1,"full_name":"灵芸黄","email":"lingyun.huang@gmail.com","telephone_number":13842681484,"sex":true,"birth_date":"1996-05-23T21:17:45.000+00:00","identity_card_number":361405197211160626}}%