线上出故障时,如果只靠客户反馈才发现,说明告警链路缺失。这里回顾用钉钉做异常告警需要哪些关键信息,以及如何把 Eureka 实例上下线也纳入告警。
为什么要做钉钉通知
一次线上故障排查下来,最让人意外的是日志里没有任何输出,问题只能等客户反馈。改进从两件事开始:统一日志输出、定义全局异常级别。
效果图:

通过全局拦截器在公共基础项目里加入 AOP 拦截,最初推送的告警包含这些信息:
告警信息
工程名:gateway
类路径:cn.z201.cloud.gateway.GatewayApplicationTest
方法名:alarm
异常信息:java.lang.IllegalAccessException
异常追踪:
cn.z201.cloud.gateway.GatewayApplicationTest.alarm(GatewayApplicationTest.java:64)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
告警刚上线时一天能推几千条,头几天大家还很紧张,过一个月就都麻木了。这说明只堆信息不够,还要能区分来源和级别。
用请求头补充客户端上下文
前端有安卓、iOS、微信小程序、Web、快应用等多个入口,后端需要识别问题来自哪个项目。做法是约定一组请求头,前端在调用时带上,告警时一并输出:
| 请求头 | 说明 |
|---|---|
Client-Business-Group-Source | 业务组来源编号 |
Client-Business-Source | 业务来源编号 |
Client-Business-Activity-Source | 特殊业务流量标识 |
Client-Env-Source | 客户端环境:1 iOS、2 Android、3 Windows |
Client-Platform-Source | 客户端平台,如手机型号、浏览器 |
Client-Start-Time | 请求时间戳 |
Client-Version-Source | 客户端版本号 |
告警随之补充了扩展信息:
告警信息
工程名:gateway
类路径:cn.z201.cloud.gateway.GatewayApplicationTest
方法名:alarm
异常信息:java.lang.IllegalAccessException
异常扩展信息: {
"Client-Business-Source": "1000",
"Client-Business-Activity-Source": "1",
"Client-Env-Source": "1",
"Client-Platform-Source": "xxx",
"Client-Version-Source": "1.0.0"
}
异常追踪:
cn.z201.cloud.gateway.GatewayApplicationTest.alarm(GatewayApplicationTest.java:64)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
跨服务透传请求头
调用链变长后,定位问题更依赖上下文。应用 A 调用应用 B,B 抛出异常,但 Spring Cloud 的 HTTP RPC 默认不会把这些请求头继续传给下游,需要在调用侧做一次透传:

先统一定义请求头常量:
/**
* @author z201.coding@gmail.com
**/
public interface HttpApiConstant {
String X_REAL_IP = "x-real-ip";
/**
* 请求头跟踪 id 名。
*/
String HTTP_HEADER_TRACE_ID = "AppTraceId";
/**
* 请求头
*/
String HTTP_TOKEN_HEADER = "Authorization";
/**
* app 租户
*/
String APP_TENANT = "Tenant";
String CLIENT_BUSINESS_GROUP_SOURCE = "Client-Business-Group-Source";
String CLIENT_BUSINESS_SOURCE = "Client-Business-Source";
String CLIENT_BUSINESS_ACTIVITY_SOURCE = "Client-Business-Activity-Source";
String CLIENT_EVN_SOURCE = "Client-Env-Source";
String CLIENT_PLATFORM_SOURCE = "Client-Platform-Source";
String CLIENT_START_TIME = "Client-Start-Time";
String CLIENT_VERSION_SOURCE = "Client-Version-Source";
}
再用 Feign 拦截器把入站请求头复制到出站请求:
/**
* @author z201.coding@gmail.com
* 自定义 Feign 拦截器
* 把请求头从应用层传到内部服务
**/
@Slf4j
@ConditionalOnClass(WebMvcConfigurer.class)
public class MdcFeignInterceptorConfig implements RequestInterceptor, HttpApiConstant {
public MdcFeignInterceptorConfig() {
log.info("Loaded Z-REST-INTERCEPTOR [V1.0.0]");
}
@Override
public void apply(RequestTemplate template) {
try {
HttpServletRequest request = ((ServletRequestAttributes) RequestContextHolder.currentRequestAttributes())
.getRequest();
String xRealIp = request.getHeader(X_REAL_IP);
String authentication = request.getHeader(HTTP_TOKEN_HEADER);
String appTraceId = request.getHeader(HTTP_HEADER_TRACE_ID);
String businessGroupSource = request.getHeader(CLIENT_BUSINESS_GROUP_SOURCE);
String clientBusinessSource = request.getHeader(CLIENT_BUSINESS_SOURCE);
String clientBusinessActivitySource = request.getHeader(CLIENT_BUSINESS_ACTIVITY_SOURCE);
String clientEnvSource = request.getHeader(CLIENT_EVN_SOURCE);
String clientPlatformSource = request.getHeader(CLIENT_PLATFORM_SOURCE);
String clientStartTime = request.getHeader(CLIENT_START_TIME);
String clientVersionSource = request.getHeader(CLIENT_VERSION_SOURCE);
template.header(HttpHeaders.ACCEPT_ENCODING, "gzip");
template.header(X_REAL_IP, xRealIp);
template.header(HTTP_TOKEN_HEADER, authentication);
template.header(CLIENT_BUSINESS_GROUP_SOURCE, businessGroupSource);
template.header(CLIENT_BUSINESS_SOURCE, clientBusinessSource);
template.header(CLIENT_BUSINESS_ACTIVITY_SOURCE, clientBusinessActivitySource);
template.header(CLIENT_EVN_SOURCE, clientEnvSource);
template.header(CLIENT_PLATFORM_SOURCE, clientPlatformSource);
template.header(CLIENT_START_TIME, clientStartTime);
template.header(CLIENT_VERSION_SOURCE, clientVersionSource);
if (log.isDebugEnabled()) {
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String name = headerNames.nextElement();
String value = request.getHeader(name);
log.debug("header {} - {}", name, value);
}
}
} catch (Exception e) {
log.error("template exception {}", e.getMessage());
}
}
}
Eureka 实例上下线告警
应用异常之外,实例退出或新实例上线同样需要通知。Eureka 暴露了注册与注销事件,监听后转发到同一个钉钉通道即可:

关键代码
@EventListener
public void listen(EurekaInstanceCanceledEvent event) {
log.error("告警通知 [{}] 服务注销 timestamp [{}] serverId [{}]",
event.getAppName(),
event.getTimestamp(),
event.getServerId());
if (!event.isReplication()) {
ServiceNotice serviceNotice = new ServiceNotice();
serviceNotice.setTitle("告警通知 服务下线");
serviceNotice.setAppName(event.getAppName());
serviceNotice.setTimestamp(event.getTimestamp());
serviceNotice.setServerId(event.getServerId());
serviceNotice.setProfiles(active);
alarmNoticeManage.createNotice(serviceNotice, "");
}
}
@EventListener
public void listen(EurekaInstanceRegisteredEvent event) {
log.info("告警通知 [{}] 服务注册 timestamp [{}] id [{}] ipAddr [{}] ",
event.getInstanceInfo().getAppName(),
event.getTimestamp(),
event.getInstanceInfo().getId(),
event.getInstanceInfo().getIPAddr()
);
if (!event.isReplication()) {
ServiceNotice serviceNotice = new ServiceNotice();
serviceNotice.setTitle("告警通知 服务上线");
serviceNotice.setAppName(event.getInstanceInfo().getAppName());
serviceNotice.setTimestamp(event.getTimestamp());
serviceNotice.setServerId(event.getInstanceInfo().getId());
serviceNotice.setProfiles(active);
alarmNoticeManage.createNotice(serviceNotice, "");
}
}
告警效果
告警通知 服务下线
2020-07-16 16:03:20 GATEWAY 192.168.31.7:gateway:9000 profiles dev1
@xxx
Eureka 源码中共有五个事件,可按需要选择监听:
EurekaServerStartedEvent - Eureka 服务端启动事件
EurekaRegistryAvailableEvent - Eureka 服务端可用事件
EurekaInstanceRegisteredEvent - Eureka 客户端服务注册事件
EurekaInstanceRenewedEvent - Eureka 客户端续约事件
EurekaInstanceCanceledEvent - Eureka 客户端下线事件
小结
AOP 全局拦截把工程名、类路径、方法与堆栈推到钉钉,请求头用 Client-Env-Source 等字段区分客户端与业务来源;跨服务要靠 Feign 拦截器透传,否则下游告警缺少上下文。Eureka 侧监听注册与注销事件,并判断 isReplication() 为 false,避免集群副本同步导致重复告警。