一背景讲故事前些天有位朋友在微信上找到我说他们的电力系统出现了内存暴涨的问题让我帮忙看下怎么回事由于不是训练营里的学员所以这个分析能力还是有点门外汉让这位朋友将dump丢给我然后我们上windbg分析。二内存暴涨分析为什么会暴涨要想知道为什么会涨很简单用老套路观察内存分布就行了首先上 !address -summary 即可。0:000 !address -summary— Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotalFree 206 7ffef5a2e000 ( 127.996 TB) 100.00% unknown 252 0f9541000 ( 3.896 GB) 93.61% 0.00%Image 1070 00c047000 ( 192.277 MB) 4.51% 0.00% Stack 195 003f80000 ( 63.500 MB) 1.49% 0.00%Heap 50 000e58000 ( 14.344 MB) 0.34% 0.00% Other 13 0001df000 ( 1.871 MB) 0.04% 0.00%TEB 65 000082000 ( 520.000 kB) 0.01% 0.00% PEB 1 000001000 ( 4.000 kB) 0.00% 0.00%— Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotalMEM_PRIVATE 522 0f9fd3000 ( 3.906 GB) 93.85% 0.00% MEM_IMAGE 1075 00c06c000 ( 192.422 MB) 4.52% 0.00%MEM_MAPPED 49 004583000 ( 69.512 MB) 1.63% 0.00%— State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotalMEM_FREE 206 7ffef5a2e000 ( 127.996 TB) 100.00% MEM_COMMIT 1360 0f1915000 ( 3.774 GB) 90.69% 0.00%MEM_RESERVE 286 018cad000 ( 396.676 MB) 9.31% 0.00%接下来使用 !eeheap -gc 观察托管堆这样就可以一分为二的则出问题。0:000 !eeheap -gcNumber of GC Heaps: 1generation 0 starts at 0x000001d1803e6ee8generation 1 starts at 0x000001d18038fc98generation 2 starts at 0x000001d080001000ephemeral segment allocation context: nonesegment begin allocated size000001d080000000 000001d080001000 000001d08fffffa0 0xfffefa0(268431264)000001d098000000 000001d098001000 000001d0a7ffff58 0xfffef58(268431192)000001d0a8000000 000001d0a8001000 000001d0b7ffff20 0xfffef20(268431136)000001d0c0000000 000001d0c0001000 000001d0cffffd80 0xfffed80(268430720)000001d0d8000000 000001d0d8001000 000001d0e7ffff28 0xfffef28(268431144)000001d0fb7f0000 000001d0fb7f1000 000001d10b7effb8 0xfffefb8(268431288)000001d10db60000 000001d10db61000 000001d11db5fe20 0xfffee20(268430880)000001d11db60000 000001d11db61000 000001d12db5fcf0 0xfffecf0(268430576)000001d12db60000 000001d12db61000 000001d13db5ff30 0xfffef30(268431152)000001d13db60000 000001d13db61000 000001d14db5ffd8 0xfffefd8(268431320)000001d14db60000 000001d14db61000 000001d15db5ffe8 0xfffefe8(268431336)000001d15db60000 000001d15db61000 000001d16db5d040 0xfffc040(268419136)000001d16db60000 000001d16db61000 000001d17db5ada0 0xfff9da0(268410272)000001d17db60000 000001d17db61000 000001d1808f97b0 0x2d987b0(47810480)Large object heap starts at 0x000001d090001000segment begin allocated size000001d090000000 000001d090001000 000001d0952f8028 0x52f7028(86994984)000001d0d0000000 000001d0d0001000 000001d0d6958340 0x6957340(110457664)Total Size: Size: 0xde9d0d70 (3734834544) bytes.GC Heap Size: Size: 0xde9d0d70 (3734834544) bytes.从卦中可以轻松的看出obviously当前是托管堆内存暴涨接下来观察托管堆。0:000 !dumpheap -statStatistics:MT Count TotalSize Class Name00007ffa6e7a1ec0 1 24 System.Collections.Generic.GenericEqualityComparer1[[System.Int64, mscorlib]] 00007ffa6dc5e1e0 1 24 System.Collections.Generic.GenericEqualityComparer1[[System.String, mscorlib]]00007ffa6dc5e110 1 24 System.Collections.Generic.GenericEqualityComparer1[[System.Int32, mscorlib]]00007ffa6dc58a98 1 24 System.CLSCompliantAttribute…00007ffa1132a1e8 482984 88869056 xxx.C_Sys_UserAppTokenModel00007ffa112f2488 602175 178243800 xxx.C_Sys_UserModel00007ffa6dc41ac8 229351 238910824 System.String[]00007ffa6dc59808 30618357 1297263358 System.String000001d0ebefbff0 4003171 1418932312 FreeTotal 41839190 objectsFragmented blocks larger than 0.5 MB:Addr Size Followed by000001d18060cca8 1.0MB 000001d180702c58 System.Byte[]000001d180702ce8 0.5MB 000001d18078a340 System.Net.SocketAddress000001d18078a3c0 1.4MB 000001d1808f9568 System.Byte[]从卦中 bottom-up 来看当前托管堆中貌似是 Free 导致的内存碎片化接下来就是往内存碎片化这边 investigation。真的是free导致的碎片化吗由于 free 的个数高达 400w 直接观察free的前后是什么这个有一点的实操难度at this point 我们发现C_Sys_UserAppTokenModel 和 C_Sys_UserModel 也高达48w和60w虽不是很高但也给我们留了是不是实体失控的模糊影子接下来祭出perfview观察这两哥们是不是来自于某个集合变量截图如下imageimage从卦中可以清晰的看到托管堆上有三个 static 变量分别吃了1G712M 和 282M尤其是1G的 s_DeviceManager 集合经过展开 perfview 是一个 ConcurrentDictionaryInt32, List 集合能吃1G的ConcurrentDictionary要么当缓存使用要么就是失控。有了这些概念之后我们再次回望托管堆上的free 的前后数据输出如下0:000 !eeheap -gcNumber of GC Heaps: 1generation 0 starts at 0x000001d1803e6ee8generation 1 starts at 0x000001d18038fc98generation 2 starts at 0x000001d080001000ephemeral segment allocation context: nonesegment begin allocated size000001d080000000 000001d080001000 000001d08fffffa0 0xfffefa0(268431264)000001d098000000 000001d098001000 000001d0a7ffff58 0xfffef58(268431192)000001d0a8000000 000001d0a8001000 000001d0b7ffff20 0xfffef20(268431136)000001d0c0000000 000001d0c0001000 000001d0cffffd80 0xfffed80(268430720)000001d0d8000000 000001d0d8001000 000001d0e7ffff28 0xfffef28(268431144)000001d0fb7f0000 000001d0fb7f1000 000001d10b7effb8 0xfffefb8(268431288)000001d10db60000 000001d10db61000 000001d11db5fe20 0xfffee20(268430880)000001d11db60000 000001d11db61000 000001d12db5fcf0 0xfffecf0(268430576)000001d12db60000 000001d12db61000 000001d13db5ff30 0xfffef30(268431152)000001d13db60000 000001d13db61000 000001d14db5ffd8 0xfffefd8(268431320)000001d14db60000 000001d14db61000 000001d15db5ffe8 0xfffefe8(268431336)000001d15db60000 000001d15db61000 000001d16db5d040 0xfffc040(268419136)000001d16db60000 000001d16db61000 000001d17db5ada0 0xfff9da0(268410272)000001d17db60000 000001d17db61000 000001d1808f97b0 0x2d987b0(47810480)Large object heap starts at 0x000001d090001000segment begin allocated size000001d090000000 000001d090001000 000001d0952f8028 0x52f7028(86994984)000001d0d0000000 000001d0d0001000 000001d0d6958340 0x6957340(110457664)Total Size: Size: 0xde9d0d70 (3734834544) bytes.GC Heap Size: Size: 0xde9d0d70 (3734834544) bytes.0:000 !dumpheap 000001d10db61000 000001d11db5fe20…000001d10db62408 00007ffa11436458 272000001d10db62518 00007ffa114365d8 48000001d10db62548 000001d0ebefbff0 422 Free000001d10db626f0 00007ffa6d54ac78 40000001d10db62718 00007ffa6dc59808 34000001d10db62740 00007ffa6dc59808 34000001d10db62768 000001d0ebefbff0 110 Free000001d10db627d8 00007ffa6d54ac78 40000001d10db62800 00007ffa6dc59808 34000001d10db62828 00007ffa6dc59808 34…0:000 !gcroot 000001d10db62740Thread 2aa0:000000a1905be300 00007ffa1112fead xxx.C_DeviceManager.b__30_0()rbp10: 000000a1905be3b0- 000001d0805121a8 xxx.C_DeviceManager- 000001d080512200 System.Collections.Concurrent.ConcurrentDictionary2[[System.Int32, mscorlib],[System.Collections.Generic.List1[[System.String, mscorlib]], mscorlib]]- 000001d142690a08 System.Collections.Concurrent.ConcurrentDictionary2Tables[[System.Int32, mscorlib],[System.Collections.Generic.List1[[System.String, mscorlib]], mscorlib]]- 000001d0912a9958 System.Collections.Concurrent.ConcurrentDictionary2Node[[System.Int32, mscorlib],[System.Collections.Generic.List1[[System.String, mscorlib]], mscorlib]][]- 000001d14250af68 System.Collections.Concurrent.ConcurrentDictionary2Node[[System.Int32, mscorlib],[System.Collections.Generic.List1[[System.String, mscorlib]], mscorlib]]- 000001d10db626f0 System.Collections.Generic.List1[[System.String, mscorlib]]- 000001d116e68958 System.String[]- 000001d10db62740 System.String最后我们到 b__30_0 中找相关的代码逻辑由于隐私关系这里就不显示了让朋友去看下 b__30_0 中为什么把 ConcurrentDictionary 搞这么大最后朋友发现是 TODB 过程中的数据堆积所致而这个堆积又引发了托管堆的碎片化有点意思。三总结