本文主要讲述一下 nginx 与 tomcat 的 502、504、503 错误及其常见的产生原因。
502 Bad Gateway : 作为网关或者代理工作的服务器尝试执行请求时,从上游服务器接收到无效的响应。
将后端服务关掉,然后向 nginx 发送请求后端接口,日志如下:
- 2017/12/22 20:45:12 [error] 1481#0: *3 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /timeout/long-write HTTP/1.1", upstream: "http://[::1]:8080/timeout//long-write", host: "localhost:8888"
504 Gateway Timeout : 作为网关或者代理工作的服务器尝试执行请求时,未能及时从上游服务器(URI 标识出的服务器,例如 HTTP、FTP、LDAP)或者辅助服务器(例如 DNS)收到响应。注意:某些代理服务器在 DNS 查询超时时会返回 400 或者 500 错误
- <html>
- <head><title>504 Gateway Time-out</title></head>
- <body bgcolor="white">
- <center><h1>504 Gateway Time-out</h1></center>
- <hr><center>openresty/1.9.15.1</center>
- </body>
- </html>
- <!-- a padding to disable MSIE and Chrome friendly error page -->
- <!-- a padding to disable MSIE and Chrome friendly error page -->
- <!-- a padding to disable MSIE and Chrome friendly error page -->
- <!-- a padding to disable MSIE and Chrome friendly error page -->
- <!-- a padding to disable MSIE and Chrome friendly error page -->
- <!-- a padding to disable MSIE and Chrome friendly error page -->
- 192.168.99.1 - -[22 / Dec / 2017 : 21 : 58 : 20 + 0800]"GET /timeout/long-resp HTTP/1.1"504 591 "-""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36""-""-"
- 2017/12/22 21:58:20 [error] 5#5: *7 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.99.1, server: , request: "GET /timeout/long-resp HTTP/1.1", upstream: "http://192.168.99.100:8080/timeout//long-resp", host: "192.168.99.100:8686"
- location / timeout / long - resp {
- proxy_connect_timeout 30;
- proxy_read_timeout 100;
- proxy_send_timeout 10;
- proxy_pass http: //192.168.99.100:8080/timeout/long-resp ;
- }
- @GetMapping("/timeout/long-resp")
- public String longResp() throws InterruptedException {
- TimeUnit.SECONDS.sleep(120);
- return "finish";
- }
服务器接受请求一直没有返回, nginx 在等待 100 秒后报 Connection timed out, 返回 504;但是后端继续执行,在第 120 秒才执行完。
503 Service Unavailable : 表示服务器当前处于暂时不可用状态,无论是有意还是无意,当服务器端处于无法应答的状态时,就会返回该状态码。其中,服务端因维护需要而停止服务属于有意的情况。而当服务器自身负载过高,处于无法响应的状态时,则属于无意的情况。另外,负载均衡器或者 web 服务器的前置机等这些地方的服务器也有可能返回 503.
- http{
- ## test 503
- limit_conn_zone $binary_remote_addr zone=addr:10m;
- server {
- listen 8686;
- location /timeout {
- limit_conn addr 1;
- proxy_connect_timeout 30;
- proxy_read_timeout 100;
- proxy_send_timeout 2;
- proxy_pass http://192.168.99.100:8080/timeout/ ;
- }
- }
- }
- 2017/12/24 20:58:29 [error] 5#5: *1473 limiting connections by zone "addr", client: 192.168.99.1, server: , request: "GET /timeout/busy HTTP/1.1", host: "192.168.99.100:8686"
- 192.168.99.1 - -[24 / Dec / 2017 : 20 : 58 : 39 + 0800]"GET /timeout/busy HTTP/1.1"503 219 "-""-""-""-"
- wrk -t12 -c200 -d100s -T60s --latency http://192.168.99.100:8686/timeout/busy
- ➜ ~ curl -i http://192.168.99.100:8686/timeout/busy
- HTTP/1.1 503 Service Temporarily Unavailable
- Server: openresty/1.9.15.1
- Date: Sun, 24 Dec 2017 12:58:26 GMT
- Content-Type: text/html
- Content-Length: 219
- Connection: keep-alive
- <html>
- <head><title>503 Service Temporarily Unavailable</title></head>
- <body bgcolor="white">
- <center><h1>503 Service Temporarily Unavailable</h1></center>
- <hr><center>openresty/1.9.15.1</center>
- </body>
- </html>
tomcat-embed-core-8.5.23-sources.jar!/org/apache/coyote/http11/Http11Processor.java
- @Override
- public SocketState service(SocketWrapperBase<?> socketWrapper)
- throws IOException {
- RequestInfo rp = request.getRequestProcessor();
- rp.setStage(org.apache.coyote.Constants.STAGE_PARSE);
- // Setting up the I/O
- setSocketWrapper(socketWrapper);
- inputBuffer.init(socketWrapper);
- outputBuffer.init(socketWrapper);
- // Flags
- keepAlive = true;
- openSocket = false;
- readComplete = true;
- boolean keptAlive = false;
- SendfileState sendfileState = SendfileState.DONE;
- while (!getErrorState().isError() && keepAlive && !isAsync() && upgradeToken == null &&
- sendfileState == SendfileState.DONE && !endpoint.isPaused()) {
- //......
- if (endpoint.isPaused()) {
- // 503 - Service unavailable
- response.setStatus(503);
- setErrorState(ErrorState.CLOSE_CLEAN, null);
- } else {
- keptAlive = true;
- // Set this every time in case limit has been changed via JMX
- request.getMimeHeaders().setLimit(endpoint.getMaxHeaderCount());
- if (!inputBuffer.parseHeaders()) {
- // We've read part of the request, don't recycle it
- // instead associate it with the socket
- openSocket = true;
- readComplete = false;
- break;
- }
- if (!disableUploadTimeout) {
- socketWrapper.setReadTimeout(connectionUploadTimeout);
- }
- }
- }
- }
只要 endpoint 的状态是 paused,则返回 503
tomcat-embed-core-8.5.23-sources.jar!/org/apache/tomcat/util/net/AbstractEndpoint.java
- /**
- * Pause the endpoint, which will stop it accepting new connections.
- */
- public void pause() {
- if (running && !paused) {
- paused = true;
- unlockAccept();
- getHandler().pause();
- }
- }
- /**
- * Resume the endpoint, which will make it start accepting new connections
- * again.
- */
- public void resume() {
- if (running) {
- paused = false;
- }
- }
这里是 endpoint 的 pause 以及 resume 方法
当请求进入 Http11Processor 的 service 方法到执行 endpoint.isPaused() 方法期间,tomcat 被 pause 了,这个时候,就会返回 503,如下:
- ➜ ~ curl -i http://localhost:8080/demo/test
- HTTP/1.1 503
- Transfer-Encoding: chunked
- Date: Sun, 24 Dec 2017 14:10:16 GMT
- Connection: close
通常是后端服务挂了或在重启
通常是请求的接口执行耗时,亦或是后端服务负载高,执行耗时
通常是 nginx 限流或后端服务 pause 进行维护
来源: https://segmentfault.com/a/1190000012595322