本篇幅只是回顾使用钉钉做异常告警需要那些关键业务信息。
为什么要做钉钉通知？
事情要从我入职上家公司说起，进入公司后把线上项目clone下来大致看了下。代码风格过于滞后、编码风格混乱。进入公司第一周就出现了线上故障，嗯。我去线上检查日志，emmmm竟然没有日志输出。这次故障是由客户反馈来的。当时我非常吃惊，大伙好像很淡定的样子，习以为常了？
想到当初面试的时候和总监的谈话，主要是带领团队落地微服务架构，看来必须大刀阔斧了。
首先想到的时候改进日志输出、定义全局异常级别，根据异常级别输出日志。
- 效果图
我们需要从钉钉里面看到那些异常信息呢？这是当时输出到钉钉的信息。通过编写全局拦截器，在公共基础项目里面添加了aop全局拦截。刚开始上线的时候钉钉一天动不动就几千个异常告警。刚开始大伙都很紧张，过了个把月大伙已经又麻木了。
告警信息
工程名：z201-gateway
类路径：cn.z201.cloud.gateway.VlinkFrameworkGatewayApplicationTest
方法名：alarm
异常信息：java.lang.IllegalAccessException
异常追踪：
cn.z201.cloud.gateway.VlinkFrameworkGatewayApplicationTest.alarm(VlinkFrameworkGatewayApplicationTest.java:64)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
这样的告警信息就够了吗？
明显这样是不够够的，前端有安卓、ios、微信小程序、web、快应用。太多前端项目了，后端需要识别出是哪里的项目出的问题。于是又改进了一次。邀请前端开发人员在HttpHeader里面增加额外参数。为了做流量区分也增加了一些参数。
{
"Content-Type":"application/json"
"Authorization":"Bearer xxxxxxxxxx" ## jwt
"Client-Business-Group-Source": "1", ## 业务组来源唯一标号
"Client-Business-Source": "1000", ## 业务来源唯一标号
"Client-Business-Activity-Source": "1", ## 查看介绍、更多 针对特殊业务流量识别
"Client-Env-Source": "1", ##客户端环境来源 1 ios 2 android 3 windows
"Client-Platform-Source": "xxx", ##客户端平台 xxx手机型号、浏览器
"Client-Start-Time": "1", ##请求时间戳
"Client-Version-Source": "1.0.0" ##客户端版本号
}
通过上面的改进告警信息完善很多。
告警信息
工程名：z201-gateway
类路径：cn.z201.cloud.gateway.VlinkFrameworkGatewayApplicationTest
方法名：alarm
异常信息：java.lang.IllegalAccessException
异常扩展信息： {
"Client-Business-Source": "1000",
"Client-Business-Activity-Source": "1",
"Client-Env-Source": "1",
"Client-Platform-Source": "xxx",
"Client-Version-Source": "1.0.0"
}
异常追踪：
cn.z201.cloud.gateway.VlinkFrameworkGatewayApplicationTest.alarm(VlinkFrameworkGatewayApplicationTest.java:64)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
分布式下面临的问题！
当调用链多的时候定位问题就有点麻烦，比如应用a调用应用b。应用b执行了异常信息直接抛出了告警信息。但是spring cloud http rpc默认是不会吧请求参数传递到后面的服务中，需要我们做下简单的扩展。
/**
* @author z201.coding@gmail.com
**/
public interface HttpApiConstant {
String X_REAL_IP = "x-real-ip";
/**
* 请求头跟踪id名。
*/
String HTTP_HEADER_TRACE_ID = "AppTraceId";
/**
* 请求头
*/
String HTTP_TOKEN_HEADER = "Authorization";
/**
* app租户
*/
String APP_TENANT = "Tenant";
String CLIENT_BUSINESS_GROUP_SOURCE = "Client-Business-Group-Source";
String CLIENT_BUSINESS_SOURCE = "Client-Business-Source";
String CLIENT_BUSINESS_ACTIVITY_SOURCE = "Client-Business-Activity-Source";
String CLIENT_EVN_SOURCE = "Client-Env-Source";
String CLIENT_PLATFORM_SOURCE = "Client-Platform-Source";
String CLIENT_START_TIME = "Client-Start-Time";
String CLIENT_VERSION_SOURCE = "Client-Version-Source";
}
/**
* @author z201.coding@gmail.com
* 自定义restTemplate拦截器
* 这里可以把一些参数从应用层传到内部服务
**/
@Slf4j
@ConditionalOnClass(WebMvcConfigurer.class)
public class MdcFeignInterceptorConfig implements RequestInterceptor, HttpApiConstant {
public MdcFeignInterceptorConfig() {
log.info("Loaded Z-REST-INTERCEPTOR [V1.0.0]");
}
@Override
public void apply(RequestTemplate template) {
try {
HttpServletRequest request = ((ServletRequestAttributes) RequestContextHolder.currentRequestAttributes())
.getRequest();
String xRealIp = request.getHeader(X_REAL_IP);
String authentication = request.getHeader(HTTP_TOKEN_HEADER);
String appTraceId = request.getHeader(HTTP_HEADER_TRACE_ID);
String businessGroupSource = request.getHeader(CLIENT_BUSINESS_GROUP_SOURCE);
String clientBusinessSource = request.getHeader(CLIENT_BUSINESS_SOURCE);
String clientBusinessActivitySource = request.getHeader(CLIENT_BUSINESS_ACTIVITY_SOURCE);
String clientEnvSource = request.getHeader(CLIENT_EVN_SOURCE);
String clientPlatformSource = request.getHeader(CLIENT_PLATFORM_SOURCE);
String clientStartTime = request.getHeader(CLIENT_START_TIME);
String clientVersionSource = request.getHeader(CLIENT_VERSION_SOURCE);
template.header(HttpHeaders.ACCEPT_ENCODING, "gzip");
template.header(X_REAL_IP, xRealIp);
template.header(HTTP_TOKEN_HEADER, authentication);
template.header(CLIENT_BUSINESS_GROUP_SOURCE, businessGroupSource);
template.header(CLIENT_BUSINESS_SOURCE, clientBusinessSource);
template.header(CLIENT_BUSINESS_ACTIVITY_SOURCE, clientBusinessActivitySource);
template.header(CLIENT_EVN_SOURCE, clientEnvSource);
template.header(CLIENT_PLATFORM_SOURCE, clientPlatformSource);
template.header(CLIENT_START_TIME, clientStartTime);
template.header(CLIENT_VERSION_SOURCE, clientVersionSource);
if (log.isDebugEnabled()) {
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String name = headerNames.nextElement();
String value = request.getHeader(name);
log.debug("header {} - {}", name, value);
}
}
} catch (Exception e) {
log.error("template exception {}",e.getMessage());
}
}
}
简单的异常钉钉告警就到这里结束了。