We have SQL server 2016 standard edition cluster setup. Active active cluster setup.
We ran into some disk errors on node two DBS02. We had replaced the failed disk and upgraded the firmware.
After upgrade and disk replacement we were not able to failover the first Instance that is SQL01 to node DBS02
The instance SQL01 is running smoothly on DBS01 (i.e node 01)
Below error is logged in SQL logs
SQL Server detected 4 sockets with 10 cores per socket and 20 logical processors per socket, 80 total logical processors; using 48 logical processors based on SQL Server licensing. This is an informational message; no
user action is required.
2020-06-09 23:02:32.09 Server SQL Server is starting at normal priority base (=7). This is an informational message only. No user action is required.
2020-06-09 23:02:32.09 Server Detected 392901 MB of RAM. This is an informational message; no user action is required.
2020-06-09 23:02:32.09 Server Using locked pages in the memory manager.
2020-06-09 23:02:32.72 Server Machine supports memory error recovery. SQL memory protection is enabled to recover from memory corruption.
2020-06-09 23:02:32.89 Server Default collation: SQL_Latin1_General_CP1_CI_AS (us_english 1033)
2020-06-09 23:02:32.89 Server Automatic soft-NUMA was enabled because SQL Server has detected hardware NUMA nodes with greater than 8 physical cores.
2020-06-09 23:02:32.91 Server Error: 17311, Severity: 16, State: 1.
2020-06-09 23:02:32.91 Server SQL Server is terminating because of fatal exception c0000005. This error may be caused by an unhandled Win32 or C++ exception, or by an access violation encountered during exception handling. Check
the SQL error log for any related stack dumps or messages. This exception forces SQL Server to shutdown. To recover from this error, restart the server (unless SQLAgent is configured to auto restart).
2020-06-09 23:02:32.91 Server Using 'dbghelp.dll' version '4.0.5'
2020-06-09 23:02:32.91 Server **Dump thread - spid = 0, EC = 0x0000000000000000
2020-06-09 23:02:32.91 Server * *******************************************************************************
2020-06-09 23:02:32.91 Server *
2020-06-09 23:02:32.91 Server * BEGIN STACK DUMP:
2020-06-09 23:02:32.91 Server * 06/09/20 23:02:32 spid 15344
2020-06-09 23:02:32.91 Server *
2020-06-09 23:02:32.91 Server * ex_handle_except encountered exception C0000005 - Server terminating
2020-06-09 23:02:32.91 Server *
2020-06-09 23:02:32.91 Server *
* MODULE LISTING:
*
2020-06-09 23:02:32.91 Server * MODULE BASE END SIZE
2020-06-09 23:02:32.91 Server * sqlservr 00007FF6F7870000 00007FF6F78D7FFF 00068000
2020-06-09 23:02:32.91 Server * ntdll 00007FFAC9800000 00007FFAC99ACFFF 001ad000
2020-06-09 23:02:32.91 Server * KERNEL32 00007FFAC7F10000 00007FFAC804EFFF 0013f000
2020-06-09 23:02:32.91 Server * KERNELBASE 00007FFAC6D70000 00007FFAC6E84FFF 00115000
2020-06-09 23:02:32.91 Server * NETAPI32 00007FFAC2F60000 00007FFAC2F75FFF 00016000
2020-06-09 23:02:32.91 Server * pdh 00007FFAC51F0000 00007FFAC523EFFF 0004f000
2020-06-09 23:02:32.91 Server * ADVAPI32 00007FFAC7BB0000 00007FFAC7C59FFF 000aa000
2020-06-09 23:02:32.91 Server * ole32 00007FFAC7560000 00007FFAC76F3FFF 00194000
2020-06-09 23:02:32.91 Server * SQLOS 00007FFAAE100000 00007FFAAE106FFF 00007000
2020-06-09 23:02:32.91 Server * sqlmin 00007FFA96E10000 00007FFA9930AFFF 024fb000
2020-06-09 23:02:32.91 Server * sqllang 00007FFA94840000 00007FFA96E09FFF 025ca000
2020-06-09 23:02:32.91 Server * sqlTsEs 00007FFA93FB0000 00007FFA9483FFFF 00890000
2020-06-09 23:02:32.91 Server * sqldk 00007FFA93AE0000 00007FFA93FA8FFF 004c9000
2020-06-09 23:02:32.91 Server * opends60 00007FFAADBF0000 00007FFAADBF8FFF 00009000
2020-06-09 23:02:32.91 Server * qds 00007FFA9BBD0000 00007FFA9BCABFFF 000dc000
2020-06-09 23:02:32.91 Server * MSVCP120 00007FFABCDB0000 00007FFABCE55FFF 000a6000
2020-06-09 23:02:32.91 Server * MSVCR120 00007FFABCE60000 00007FFABCF4EFFF 000ef000
2020-06-09 23:02:32.91 Server * netutils 00007FFAC2EF0000 00007FFAC2EFBFFF 0000c000
2020-06-09 23:02:32.91 Server * srvcli 00007FFAC6500000 00007FFAC6525FFF 00026000
2020-06-09 23:02:32.91 Server * wkscli 00007FFAC2ED0000 00007FFAC2EE6FFF 00017000
2020-06-09 23:02:32.91 Server * msvcrt 00007FFAC6FC0000 00007FFAC7069FFF 000aa000
2020-06-09 23:02:32.91 Server * RPCRT4 00007FFAC7780000 00007FFAC78BFFFF 00140000
2020-06-09 23:02:32.91 Server * sechost 00007FFAC7EB0000 00007FFAC7F08FFF 00059000
2020-06-09 23:02:32.91 Server * combase 00007FFAC7070000 00007FFAC727FFFF 00210000
2020-06-09 23:02:32.91 Server * GDI32 00007FFAC7280000 00007FFAC73CBFFF 0014c000
2020-06-09 23:02:32.91 Server * USER32 00007FFAC73D0000 00007FFAC7546FFF 00177000
2020-06-09 23:02:32.91 Server * CRYPT32 00007FFAC6AE0000 00007FFAC6CBEFFF 001df000
2020-06-09 23:02:32.91 Server * Secur32 00007FFAC5C70000 00007FFAC5C7BFFF 0000c000
2020-06-09 23:02:32.91 Server * WS2_32 00007FFAC97A0000 00007FFAC97F9FFF 0005a000
2020-06-09 23:02:32.91 Server * WINHTTP 00007FFAC0DF0000 00007FFAC0EB8FFF 000c9000
2020-06-09 23:02:32.91 Server * bcrypt 00007FFAC63D0000 00007FFAC63F5FFF 00026000
2020-06-09 23:02:32.91 Server * OLEAUT32 00007FFAC6EB0000 00007FFAC6F76FFF 000c7000
2020-06-09 23:02:32.91 Server * ODBC32 00007FFABC8E0000 00007FFABC9A1FFF 000c2000
2020-06-09 23:02:32.91 Server * USERENV 00007FFAC5EB0000 00007FFAC5ED0FFF 00021000
2020-06-09 23:02:32.91 Server * AUTHZ 00007FFAC5B80000 00007FFAC5BC7FFF 00048000
2020-06-09 23:02:32.91 Server * WINTRUST 00007FFAC6A80000 00007FFAC6AD0FFF 00051000
2020-06-09 23:02:32.91 Server * XmlLite 00007FFAC34E0000 00007FFAC3518FFF 00039000
2020-06-09 23:02:32.91 Server * dhcpcsvc 00007FFAC2B20000 00007FFAC2B39FFF 0001a000
2020-06-09 23:02:32.91 Server * WINMM 00007FFABC770000 00007FFABC791FFF 00022000
2020-06-09 23:02:32.91 Server * SspiCli 00007FFAC6A00000 00007FFAC6A2DFFF 0002e000
2020-06-09 23:02:32.91 Server * MSASN1 00007FFAC69E0000 00007FFAC69F0FFF 00011000
2020-06-09 23:02:32.91 Server * NSI 00007FFAC78C0000 00007FFAC78C8FFF 00009000
2020-06-09 23:02:32.91 Server * profapi 00007FFAC6930000 00007FFAC6944FFF 00015000
2020-06-09 23:02:32.91 Server * WINMMBASE 00007FFABC740000 00007FFABC769FFF 0002a000
2020-06-09 23:02:32.91 Server * LOGONCLI 00007FFAC5EE0000 00007FFAC5F1EFFF 0003f000
2020-06-09 23:02:32.91 Server * cfgmgr32 00007FFAC6A30000 00007FFAC6A7EFFF 0004f000
2020-06-09 23:02:32.91 Server * DEVOBJ 00007FFAC5760000 00007FFAC5787FFF 00028000
2020-06-09 23:02:32.91 Server * SAMCLI 00007FFAC1020000 00007FFAC1036FFF 00017000
2020-06-09 23:02:32.91 Server * SOPHOS~1 00007FFAC6870000 00007FFAC68ABFFF 0003c000
2020-06-09 23:02:32.91 Server * PSAPI 00007FFAC7550000 00007FFAC7556FFF 00007000
2020-06-09 23:02:32.91 Server * kernel.appcore 00007FFAC5580000 00007FFAC558AFFF 0000b000
2020-06-09 23:02:32.91 Server * CRYPTBASE 00007FFAC6760000 00007FFAC676AFFF 0000b000
2020-06-09 23:02:32.91 Server * bcryptPrimitives 00007FFAC66F0000 00007FFAC6752FFF 00063000
2020-06-09 23:02:32.91 Server * instapi130 00007FFABC810000 00007FFABC81FFFF 00010000
2020-06-09 23:02:32.91 Server * CRYPTSP 00007FFAC6180000 00007FFAC619FFFF 00020000
2020-06-09 23:02:32.91 Server * rsaenh 00007FFAC5DC0000 00007FFAC5DF5FFF 00036000
2020-06-09 23:02:32.91 Server * imagehlp 00007FFAC7700000 00007FFAC7715FFF 00016000
2020-06-09 23:02:32.91 Server * ncrypt 00007FFAC63A0000 00007FFAC63C4FFF 00025000
2020-06-09 23:02:32.91 Server * NTASN1 00007FFAC6360000 00007FFAC6396FFF 00037000
2020-06-09 23:02:32.91 Server * gpapi 00007FFAC5AD0000 00007FFAC5AF3FFF 00024000
2020-06-09 23:02:32.91 Server * CLUSAPI 00007FFABA040000 00007FFABA0ADFFF 0006e000
2020-06-09 23:02:32.91 Server * cryptdll 00007FFAC62A0000 00007FFAC62B9FFF 0001a000
2020-06-09 23:02:32.91 Server * RESUTILS 00007FFAB2AA0000 00007FFAB2AEBFFF 0004c000
2020-06-09 23:02:32.91 Server * cscapi 00007FFAB8A00000 00007FFAB8A11FFF 00012000
2020-06-09 23:02:32.91 Server * sqlevn70 00007FFA937F0000 00007FFA93ADFFFF 002f0000
2020-06-09 23:02:32.91 Server * VERSION 00007FFAC4EB0000 00007FFAC4EB9FFF 0000a000
2020-06-09 23:02:32.91 Server * dbghelp 00000000723C0000 0000000072555FFF 00196000
2020-06-09 23:02:32.91 Server *
* PROCESSOR SPECIFIC CONTEXT:
*
2020-06-09 23:02:32.91 Server * P1Home: 00007FFAC7898A60: 6C894810245C8948 8348564157561824 0000B0998B4830EC 00000160B18B4800 48000000A8B18B4C DB8548F98B48EA8B
2020-06-09 23:02:32.91 Server * P2Home: 0000000000000000:
2020-06-09 23:02:32.91 Server * P3Home: 0000000000000000:
2020-06-09 23:02:32.91 Server * P4Home: 0000000000000000:
2020-06-09 23:02:32.91 Server * P5Home: 0000000000000000:
2020-06-09 23:02:32.91 Server * P6Home: 0000000000000000:
2020-06-09 23:02:32.91 Server * ContextFlags: 000000000010004F:
2020-06-09 23:02:32.91 Server * MxCsr: 0000000000001FA0:
2020-06-09 23:02:32.91 Server * SegCs: 0000000000000033:
2020-06-09 23:02:32.91 Server * SegDs: 000000000000002B:
2020-06-09 23:02:32.91 Server * SegEs: 000000000000002B:
2020-06-09 23:02:32.91 Server * SegFs: 0000000000000053:
2020-06-09 23:02:32.91 Server * SegGs: 000000000000002B:
2020-06-09 23:02:32.91 Server * SegSs: 000000000000002B:
2020-06-09 23:02:32.91 Server * EFlags: 0000000000000206:
2020-06-09 23:02:32.91 Server * Rax: 0000005411949178: 00000000C0000100 0000005410B50CC0 000000540000002C 0000000000000020 000000000000003F 0000000000000001
2020-06-09 23:02:32.91 Server * Rcx: 000000000000079A:
2020-06-09 23:02:32.91 Server * Rdx: 00007FFAC983BE9D: C68B413B34894666 F88348FFFFFF00E9 7824448B48197301 000023BE41C1FF48 E9C68B41088948C0 37894566FFFFFEE1
2020-06-09 23:02:32.91 Server * Rbx: 0000000000000000:
2020-06-09 23:02:32.91 Server * Rsp: 00000054119496B0: 0000000000000000 0000000000000000 0000000000000000 00007FFA93B4C140 00000000000042AC 0000000000000000
2020-06-09 23:02:32.91 Server * Rbp: 0000000000000000:
2020-06-09 23:02:32.91 Server * Rsi: 0000000000000000:
2020-06-09 23:02:32.91 Server * Rdi: 00007FFA93B4C140: 6C894808245C8948 5718247489481024 8B48F18B30EC8348 330019EAED0D8BEA 480011122915FFDB F88039481C74C085
2020-06-09 23:02:32.91 Server * R8: 00000000C0000100:
2020-06-09 23:02:32.91 Server * R9: 0000005410B50CC0: 0000000000900000 0000000000000000 0000005410B4E8B0 0000005410B50000 000000200000010A 0000000000000000
2020-06-09 23:02:32.91 Server * R10: 000000540000002C:
2020-06-09 23:02:32.91 Server * R11: 0000000000000020:
2020-06-09 23:02:32.91 Server * R12: 000000000000003F:
2020-06-09 23:02:32.91 Server * R13: 0000000000000001:
2020-06-09 23:02:32.91 Server * R14: 0000000000000000:
2020-06-09 23:02:32.91 Server * R15: 0000000000000000:
2020-06-09 23:02:32.91 Server * Rip: 00007FFAC6D788DC: 000000C0248C8B48 FFFF8B54E8CC3348 C3000000D8C48148 9090909090909090 38418B4D28EC8348 11E8D18B49CA8B48
2020-06-09 23:02:32.91 Server * *******************************************************************************
2020-06-09 23:02:32.91 Server * -------------------------------------------------------------------------------
2020-06-09 23:02:32.91 Server * Short Stack Dump
2020-06-09 23:02:32.92 Server 00007FFAC6D788DC Module(KERNELBASE+00000000000088DC)
2020-06-09 23:02:32.92 Server 00007FFA9587828E Module(sqllang+000000000103828E)
2020-06-09 23:02:32.92 Server 00007FFA9587C009 Module(sqllang+000000000103C009)
2020-06-09 23:02:32.92 Server 00007FFA94DA8486 Module(sqllang+0000000000568486)
2020-06-09 23:02:32.93 Server 00007FFA93B4BC62 Module(sqldk+000000000006BC62)
2020-06-09 23:02:32.93 Server 00007FFAC6E516D2 Module(KERNELBASE+00000000000E16D2)
2020-06-09 23:02:32.93 Server 00007FFAC989F173 Module(ntdll+000000000009F173)
2020-06-09 23:02:32.93 Server 00007FFAC9881DD6 Module(ntdll+0000000000081DD6)
2020-06-09 23:02:32.93 Server 00007FFAC989345D Module(ntdll+000000000009345D)
2020-06-09 23:02:32.93 Server 00007FFAC9854937 Module(ntdll+0000000000054937)
2020-06-09 23:02:32.93 Server 00007FFAC98925EA Module(ntdll+00000000000925EA)
2020-06-09 23:02:32.93 Server 00007FFA93C25B16 Module(sqldk+0000000000145B16)
2020-06-09 23:02:32.93 Server 00007FFA93C252C2 Module(sqldk+00000000001452C2)
2020-06-09 23:02:32.93 Server 00007FFA93C250CB Module(sqldk+00000000001450CB)
2020-06-09 23:02:32.93 Server 00007FFA95EDE27A Module(sqllang+000000000169E27A)
2020-06-09 23:02:32.93 Server 00007FFA95E5DAB6 Module(sqllang+000000000161DAB6)
2020-06-09 23:02:32.93 Server 00007FF6F787D1C8 Module(sqlservr+000000000000D1C8)
2020-06-09 23:02:32.93 Server 00007FF6F787EEB3 Module(sqlservr+000000000000EEB3)
2020-06-09 23:02:32.93 Server 00007FF6F78784F8 Module(sqlservr+00000000000084F8)
2020-06-09 23:02:32.94 Server 00007FFAC7EB5ADA Module(sechost+0000000000005ADA)
2020-06-09 23:02:32.94 Server 00007FFAC7F113D2 Module(KERNEL32+00000000000013D2)
2020-06-09 23:02:32.94 Server 00007FFAC98154F4 Module(ntdll+00000000000154F4)
2020-06-09 23:02:32.94 Server Stack Signature for the dump is 0x000000000ADDB46F
2020-06-09 23:02:32.94 Server Unable to create dump because SQLDUMPER library is not available.
2020-06-09 23:02:32.94 Server SQL Server shutdown has been initiated