一背景讲故事说来也奇怪最近分析了好几例内存暴涨事故这不又来了哈哈今天再给大家带来一份非托管内存泄露导致的程序生产故障而且是部署在Linux上.NET程序。前些天有位朋友找到我自己有一定的分析能力发现程序是非托管内存泄露但能力有限暂时也不知道怎么弄让我帮忙看下把dump也丢给我了既然dump有了那就开始分析吧。二内存暴涨分析为什么会暴涨虽然朋友说了是非托管内存泄露但我们还是要相信数据使用 !maddress 观察即可。0:000 !maddress±------------------------------------------------------------------------| Memory Type | Count | Size | Size (bytes) |±------------------------------------------------------------------------| PAGE_READWRITE | 2,465 | 3.64gb | 3,913,596,928 || GCHeap | 9 | 1.45gb | 1,557,557,248 || Stack | 63 | 972.26mb | 1,019,490,304 || Image | 1,413 | 287.81mb | 301,786,112 || HighFrequencyHeap | 1,722 | 107.54mb | 112,758,784 || LowFrequencyHeap | 903 | 66.32mb | 69,545,984 || LoaderCodeHeap | 35 | 61.00mb | 63,963,136 || HostCodeHeap | 85 | 19.97mb | 20,942,848 || ResolveHeap | 3 | 1.09mb | 1,142,784 || IndirectionCellHeap | 9 | 536.00kb | 548,864 || StubHeap | 8 | 460.00kb | 471,040 || DispatchHeap | 2 | 452.00kb | 462,848 || CacheEntryHeap | 7 | 420.00kb | 430,080 || LookupHeap | 5 | 272.00kb | 278,528 || PAGE_READONLY | 125 | 263.00kb | 269,312 || PAGE_EXECUTE_WRITECOPY | 6 | 232.00kb | 237,568 || PAGE_EXECUTE_READ | 2 | 8.00kb | 8,192 |±------------------------------------------------------------------------| [TOTAL] | 6,862 | 6.58gb | 7,063,490,560 |±------------------------------------------------------------------------从卦中我们看到程序总计吃了 6.58G 内存其中 PAGE_READWRITE 3.64G很显然这是赤裸裸的非托管内存泄露说实话Linux上的非托管内存泄露不是很好弄接下来我们的研究方向在哪里呢在如今AI横行的时代可以用一个小技巧那就是用AI将 2465个 PAGE_READWRITE 做一个 groupby 操作寻找size的特征这里取 top5 的记录截图如下image从卦中可以清晰的看到主体上是被 50~200MB 这个区间吃掉了接下来我们随便挑选几个观察其中内容截图如下image从卦中可以看到大量的 lambda_methodxxx 的字样看起来和动态代码生成有关系但还不是那么清晰。lambda_method 为啥这么多不管怎么说数字这么大了冥冥之中应该是一种失控熟悉C# 的朋友应该知道这个应该是 C# 通过 DynamicMethod 类似的机制动态生成的方法所以我们的接下来的注意力应该投向托管堆了使用 !dumpheap -stat 观察托管堆。0:000 !dumpheap -stat…7f6cee5e6360 105,288 10,949,952 System.Reflection.RuntimeMethodInfo7f6cf0943da8 81,261 11,701,584 System.Reflection.Emit.DynamicILGenerator7f6cfc273e40 1,428 11,732,448 IndexItem[]7f6cf883b510 13,189 12,785,440 System.Collections.Concurrent.ConcurrentDictionarySystem.String, System.ObjectNode[]7f6cf9c3aaa0 132,063 13,734,552 OracleInternal.Common.ColumnDescribeInfo7f6cfa1f54b0 20,424 16,314,756 Oracle.ManagedDataAccess.Client.OracleParameterStatus[]7f6cf97f7cb8 55,844 28,145,376 Oracle.ManagedDataAccess.Client.OracleConnection7f6ceed94548 21,310 32,195,096 System.Byte[][]7f6cedb852a8 1,445,516 34,692,384 System.Object7f6cff21ea98 728,162 40,777,072 SkiaSharp.SKString7f6cf561ca90 1,343,650 42,996,800 Microsoft.Win32.SafeHandles.SafeEvpCipherCtxHandle7f6cedb8b110 144,685 54,552,728 System.Object[]7f6cf0cd1110 1,878,511 60,112,352 Microsoft.Win32.SafeHandles.SafeWaitHandle55ad23d21ab0 58,648 70,742,904 Free7f6cedc38080 141,177 100,202,360 System.Int32[]7f6cedc3d2e0 2,827,739 188,196,546 System.String7f6cee5ea3a0 3,787,714 280,567,631 System.Byte[]Total 18,171,140 objects, 1,533,484,720 bytesFragmented blocks larger than 0.5 MB:Address Size Followed By7f6a1d47cb18 3,359,344 7f6a1d7b0d88 System.Byte[]7f6a1d81c080 1,151,408 7f6a1d935230 System.IO.Compression.ZLibNativeZLibStreamHandle看到卦中的 DynamicILGenerator10.5w 的时候猜想再次被印证看样子非托管内存中都是它投放的接下来观察 DynamicILGenerator 的引用根看看到底咋回事。0:000 !dumpheap -mt 7f6cf0943da8Address MT Size…7f6cc7f7a3f0 7f6cf0943da8 1447f6cc7f81168 7f6cf0943da8 1447f6cc7f81598 7f6cf0943da8 1447f6cc7f81988 7f6cf0943da8 1447f6cc7f83308 7f6cf0943da8 1447f6cc7f83ff8 7f6cf0943da8 1447f6cc7f8ceb8 7f6cf0943da8 144Statistics:MT Count TotalSize Class Name7f6cf0943da8 81,261 11,701,584 System.Reflection.Emit.DynamicILGeneratorTotal 81,261 objects, 11,701,584 bytes0:000 !gcroot 7f6cc7f7a3f0Found 0 unique roots.0:000 !gcroot 7f6b67d00990Found 0 unique roots.0:000 !gcroot 7f6cc7f83ff8Found 0 unique roots.从卦中看比较奇怪的是为什么这几个 DynamicILGenerator 样本都没有引用根呢这里又藏了什么玄机呢DynamicILGenerator为什么无引用根如果你有大量的dump分析经验我相信你马上就会想到一个东西对它就是 finalizequeue因为有析构但未被回收自然就想到了它输出如下0:000 !fq -statSyncBlocks to be cleaned up: 0Free-Threaded Interfaces to be released: 0MTA Interfaces to be released: 0STA Interfaces to be released: 0Heap 0generation 0 has 25 objects (7f6b29d58238-7f6b29d58300)generation 1 has 90,693 objects (7f6b29ca7010-7f6b29d58238)generation 2 has 0 objects (7f6b29ca7010-7f6b29ca7010)Ready for finalization 4,088,657 objects (7f6b29d58488-7f6b2bc89f10)Statistics for all finalizable objects (including all objects ready for finalization):这一卦真的太灵验了从尼玛 Ready for finalization 4,088,657 objects 中发现有 408w 的对象在freachable中等待释放看样子FinalizerThread 有点麻烦了。FinalizerThread 怎么了不管怎么说直觉告诉我马上就要真相大白了有点抑制不住内心的喜悦接下来切过去观察 FinalizerThread 的调用栈输出如下0:000 ~~[11d83]slibpthread_2_170xbde2:00007f6d6816ede2 4989c6 mov r14,rax0:005 kChild-SP RetAddr Call Site00 00007f6d644892e0 00007f6d67143fd6 libpthread_2_170xbde201 00007f6d64489330 00007f6d67143ca1 libcoreclr!CorUnix::CPalSynchronizationManager::ThreadNativeWait0x126 [/__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp 488]02 00007f6d64489390 00007f6d67148e39 libcoreclr!CorUnix::CPalSynchronizationManager::BlockThread0x1d1 [/__w/1/s/src/coreclr/pal/src/synchmgr/synchmanager.cpp 307]03 (Inline Function) ---------------- libcoreclr!CorUnix::InternalSleepEx0x61 [/__w/1/s/src/coreclr/pal/src/synchmgr/wait.cpp 850] 04 00007f6d644893f0 00007f6d66db49f7 libcoreclr!SleepEx0x99 [/__w/1/s/src/coreclr/pal/src/synchmgr/wait.cpp 285] 05 00007f6d64489440 00007f6d66e0c53b libcoreclr!Thread::UserSleep0x147 [/__w/1/s/src/coreclr/vm/threads.cpp 4264] 06 00007f6d64489490 00007f6cfbde865a libcoreclr!ThreadNative::Sleep0xbb [/__w/1/s/src/coreclr/vm/comsynchronizable.cpp 471] 07 00007f6d644895f0 00007f6cfea03c0b System_Private_CoreLib!System.Threading.Thread.Sleep0x1a [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Thread.cs 357] 08 00007f6d64489620 00007f6cfea03ac9 xxxx_7f6cb4034000! 09 00007f6d64489660 00007f6d66fbc277 xxxx_7f6cb4034000! 0a 00007f6d64489700 00007f6d66df1dee libcoreclr!CallDescrWorkerInternal0x7c [/__w/1/s/src/coreclr/pal/inc/unixasmmacrosamd64.inc 854] 0b (Inline Function) ---------------- libcoreclr!CallDescrWorkerWithHandler0x59 [/__w/1/s/src/coreclr/vm/callhelpers.cpp 67]0c 00007f6d64489720 00007f6d66d19632 libcoreclr!DispatchCallSimple0xfe [/__w/1/s/src/coreclr/vm/callhelpers.cpp 220]0d 00007f6d644897b0 00007f6d66e02a6a libcoreclr!ExceptionNotifications::DeliverExceptionNotification0x420e 00007f6d64489800 00007f6d66e0269a libcoreclr!InvokeUnhandledSwallowing0xaa [/__w/1/s/src/coreclr/vm/comdelegate.cpp 3227]0f 00007f6d64489880 00007f6d66cbb5c3 libcoreclr!DistributeUnhandledExceptionReliably0x32a [/__w/1/s/src/coreclr/vm/comdelegate.cpp 3291]10 00007f6d64489aa0 00007f6d66cbb30f libcoreclr!AppDomain::RaiseUnhandledExceptionEvent0x93 [/__w/1/s/src/coreclr/vm/appdomain.cpp 4328]11 00007f6d64489b10 00007f6d66d160ad libcoreclr!AppDomain::OnUnhandledException0xdf [/__w/1/s/src/coreclr/vm/appdomain.cpp 4270]12 00007f6d64489b90 00007f6d66d14ebd libcoreclr!NotifyAppDomainsOfUnhandledException0xdd13 (Inline Function) ---------------- libcoreclr!InternalUnhandledExceptionFilter_Worker(_EXCEPTION_POINTERS*)::$_3::operator() const0xfc [/__w/1/s/src/coreclr/vm/excep.cpp 4815] 14 00007f6d64489c10 00007f6d66db7045 libcoreclr!InternalUnhandledExceptionFilter_Worker0x26d [/__w/1/s/src/coreclr/vm/excep.cpp 4896] 15 (Inline Function) ---------------- libcoreclr!ThreadBaseRedirectingFilter0xc [/__w/1/s/src/coreclr/vm/threads.cpp 7427]16 (Inline Function) ---------------- libcoreclr!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_6::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const::{lambda(PAL_SEHException)#1}::operator() const0x16 [/__w/1/s/src/coreclr/vm/threads.cpp 7525] 17 (Inline Function) ---------------- libcoreclr!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*):_6::operator() const0x5d2 [/__w/1/s/src/coreclr/vm/threads.cpp 7525]18 00007f6d64489cc0 00007f6d66db71cd libcoreclr!ManagedThreadBase_DispatchOuter0x665 [/__w/1/s/src/coreclr/vm/threads.cpp 7549]19 (Inline Function) ---------------- libcoreclr!ManagedThreadBase_NoADTransition0x18 [/__w/1/s/src/coreclr/vm/threads.cpp 7593] 1a 00007f6d64489de0 00007f6d66e3b648 libcoreclr!ManagedThreadBase::FinalizerBase0x2d [/__w/1/s/src/coreclr/vm/threads.cpp 7619] 1b 00007f6d64489e10 00007f6d67150a2e libcoreclr!FinalizerThread::FinalizerThreadStart0x58 [/__w/1/s/src/coreclr/vm/finalizerthread.cpp 391] 1c 00007f6d64489e30 00007f6d6816aea5 libcoreclr!CorUnix::CPalThread::ThreadEntry0x24e [/__w/1/s/src/coreclr/pal/src/thread/thread.cpp 1865] 1d 00007f6d64489ee0 00007f6d6746fb0d libpthread_2_170x7ea5 1e 00007f6d64489f80 ffffffffffffffff libc_2_170xfeb0d 1f 00007f6d64489f88 0000000000000000 0xffffffffffffffff尼玛这一卦打的真离谱怎么终结器线程在 Sleep 里 接下来根据 !ip2md 00007f6cfea03ac9 将代码导出来截图如下image啥要暂停1h这代码真的有点看不懂了有知道的大牛看看这个逻辑咋样
)
)
