神秘的40毫秒延迟与 TCP_NODELAY

最近的业余时间几乎全部献给 breeze 这个多年前挖 下的大坑—— 一个异步 HTTP Server。努力没有白费，项目已经逐渐成型了， 基本的框架已经有了，一个静态 文件模块也已经实现了。

写 HTTP Server，不可免俗地一定要用 ab 跑一下性能，结果一跑不打紧，出现了一个困扰了我好几天的问题：神秘的 40ms 延迟。

Table of Contents

• 1 现象
• 2 背后的原因
• 3 为什么只有 Write-Write-Read 时才会出问题
• 4 解决方案
  • 4.1 优化协议
  • 4.2 开启 TCP_NODELAY

1 现象

现象是这样的，首先看我用 ab 不加 -k 选项的结果：

[~/dev/personal/breeze]$ /usr/sbin/ab  -c 1 -n 10 http://127.0.0.1:8000/styles/shThemeRDark.css
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software:        breeze/0.1.0
Server Hostname:        127.0.0.1
Server Port:            8000

Document Path:          /styles/shThemeRDark.css
Document Length:        127 bytes

Concurrency Level:      1
Time taken for tests:   0.001 seconds
Complete requests:      10
Failed requests:        0
Write errors:           0
Total transferred:      2700 bytes
HTML transferred:       1270 bytes
Requests per second:    9578.54 [#/sec] (mean)
Time per request:       0.104 [ms] (mean)
Time per request:       0.104 [ms] (mean, across all concurrent requests)
Transfer rate:          2525.59 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     0    0   0.0      0       0
Waiting:        0    0   0.0      0       0
Total:          0    0   0.1      0       0

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%      0 (longest request)

很好，不超过 1ms 的响应时间。但一旦我加上了 -k 选项启用 HTTP Keep-Alive，结果就变成了这样：

[~/dev/personal/breeze]$ /usr/sbin/ab -k  -c 1 -n 10 http://127.0.0.1:8000/styles/shThemeRDark.css
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done

Server Software:        breeze/0.1.0
Server Hostname:        127.0.0.1
Server Port:            8000

Document Path:          /styles/shThemeRDark.css
Document Length:        127 bytes

Concurrency Level:      1
Time taken for tests:   0.360 seconds
Complete requests:      10
Failed requests:        0
Write errors:           0
Keep-Alive requests:    10
Total transferred:      2750 bytes
HTML transferred:       1270 bytes
Requests per second:    27.75 [#/sec] (mean)
Time per request:       36.041 [ms] (mean)
Time per request:       36.041 [ms] (mean, across all concurrent requests)
Transfer rate:          7.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1   36  12.4     40      40
Waiting:        0    0   0.2      0       1
Total:          1   36  12.4     40      40

Percentage of the requests served within a certain time (ms)
  50%     40
  66%     40
  75%     40
  80%     40
  90%     40
  95%     40
  98%     40
  99%     40
 100%     40 (longest request)

40ms 啊！这可是访问本机上的 Server 啊，才 1 个连接啊！太奇怪了吧！祭出 神器 strace，看看到底是什么情况：

15:37:47.493170 epoll_wait(3, {}, 1024, 0) = 0
15:37:47.493210 readv(5, [{"GET /styles/shThemeRDark.css HTT"..., 10111}, {"GET /styles/shThemeRDark.css HTT"..., 129}], 2) = 129
15:37:47.493244 epoll_wait(3, {}, 1024, 0) = 0
15:37:47.493279 write(5, "HTTP/1.0 200 OK\r\nContent-Type: t"..., 148) = 148
15:37:47.493320 write(5, "