SQL Server 2017 Dump RaiseInconsistencyError 0x5ed.
Applies to: SQL Server 2017 VM on Azure.
Date created: July 22, 2019.
Problem Description.
An Azure SQL VM customer told me the system drive of their SQL Server 2017 VM
ran out of space. When I examined the server I found on the SQL Server Log
folder (C:\PROGRAM FILES\MICROSOFT SQL SERVER\MSSQL14.MSSQLSERVER\MSSQL\LOG) SQL
Server dump files being generated by SQL Server 2017 and consuming hundreds of
gigabytes of disk space. The files I found were named like:
SQLDump10000.log
(64 KB)
SQLDump10000.mdmp (12,238 KB)
SQLDump10000.txt
(12,640 KB)
SQLDump10001.log
(64 KB)
SQLDump10001.mdmp (13,984 KB)
SQLDump10001.txt
(64 KB)
Using
Windows Debugging Tools I started to analyze those dump files and found
the following:
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
KEY_VALUES_STRING: 1
Key : Timeline.OS.Boot.DeltaSec
Value: 156124
Key : Timeline.Process.Start.DeltaSec
Value: 155574
PROCESSES_ANALYSIS: 1
SERVICE_ANALYSIS: 1
STACKHASH_ANALYSIS: 1
TIMELINE_ANALYSIS: 1
Timeline: !analyze.Start
Name: <blank>
Time: 2019-07-13T20:22:25.894Z
Diff: 1703905894 mSec
Timeline: Dump.Current
Name: <blank>
Time: 2019-06-24T03:04:00.0Z
Diff: 0 mSec
Timeline: Process.Start
Name: <blank>
Time: 2019-06-22T07:51:06.0Z
Diff: 155574000 mSec
Timeline: OS.Boot
Name: <blank>
Time: 2019-06-22T07:41:56.0Z
Diff: 156124000 mSec
DUMP_CLASS: 2
DUMP_QUALIFIER: 400
CONTEXT: (.ecxr)
rax=005c004c00510053 rbx=0000000080004005 rcx=005c0047004f004c
rdx=00000052b9e70000 rsi=0000000000040440 rdi=00007ff785c82a78
rip=00007ffb301f4c48 rsp=00000052b9e7b1f0 rbp=00000000d2b3e269
r8=000001da8b07a8c8 r9=0000000000000000 r10=00007ffb171943ae
r11=0000000000000034 r12=00007ffb19e3ff38 r13=00000000d2b3e269
r14=00000000000025d8 r15=00007ff785c82a78
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
KERNELBASE!RaiseException+0x68:
00007ffb`301f4c48 488b8c24c0000000 mov rcx,qword ptr [rsp+0C0h]
ss:00000052`b9e7b2b0=00005a65ef98086f
Resetting default scope
FAULTING_IP:
sqllang!SQLDumperLibraryInvoke+1cd
00007ffb`182859ad 448bf0 mov r14d,eax
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffb182859ad
(sqllang!SQLDumperLibraryInvoke+0x00000000000001cd)
ExceptionCode: 000042ac
ExceptionFlags: 00000000
NumberParameters: 0
DEFAULT_BUCKET_ID: APPLICATION_FAULT
PROCESS_NAME: sqlservr.exe
ERROR_CODE: (NTSTATUS) 0x42ac - <Unable to get error code text>
EXCEPTION_CODE: (Win32) 0x42ac (17068) - <Unable to get error code text>
EXCEPTION_CODE_STR: 42ac
WATSON_BKT_PROCSTAMP: 5cdced6b
WATSON_BKT_PROCVER: 2017.140.3162.1
PROCESS_VER_PRODUCT: Microsoft SQL Server
WATSON_BKT_MODULE: sqllang.dll
WATSON_BKT_MODSTAMP: 5cdceefa
WATSON_BKT_MODOFFSET: 3459ad
WATSON_BKT_MODVER: 2017.140.3162.1
MODULE_VER_PRODUCT: Microsoft SQL Server
BUILD_VERSION_STRING: 10.0.14393.2969 (rs1_release.190503-1820)
MODLIST_WITH_TSCHKSUM_HASH: 9d93f5e71e5b92a779937ea25ea10a6011d7c326
MODLIST_SHA1_HASH: e1638aafdc046631c288b30dcf901fe0f6a96c42
COMMENT: Stack Trace
NTGLOBALFLAG: 0
PROCESS_BAM_CURRENT_THROTTLED: 0
PROCESS_BAM_PREVIOUS_THROTTLED: 0
DUMP_FLAGS: 800288c
DUMP_TYPE: 1
MISSING_CLR_SYMBOL: 0
ANALYSIS_SESSION_HOST: SERVERNAME
ANALYSIS_SESSION_TIME: 07-13-2019 16:22:25.0894
ANALYSIS_VERSION: 10.0.18362.1 amd64fre
MANAGED_CODE: 1
MANAGED_ENGINE_MODULE: clr
MANAGED_ANALYSIS_PROVIDER: SOS
MANAGED_THREAD_ID: 25d8
THREAD_ATTRIBUTES:
OS_LOCALE: ENU
BUGCHECK_STR: APPLICATION_FAULT
PRIMARY_PROBLEM_CLASS: APPLICATION_FAULT
PROBLEM_CLASSES:
ID: [0n320]
Type: [APPLICATION_FAULT]
Class: Primary
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
Name: Add
Data: Omit
PID: [Unspecified]
TID: [Unspecified]
Frame: [0]
LAST_CONTROL_TRANSFER: from 00007ff785c3000c to 00007ffb301f4c48
STACK_TEXT:
00000052`b9e7b1f0 00007ff7`85c3000c : 00000000`00010000 00007fff`fffeffff
00000052`b9e7b344 00007ffb`275b3b04 : KERNELBASE!RaiseException+0x68
00000052`b9e7b2d0 00007ffb`182859ad : 00007ff7`85c82a78 00007ff7`85c30b41
00000052`b9e6a000 00007ffb`33cd8190 : sqlservr!CDmpDump::Dump+0x4c
00000052`b9e7b310 00007ffb`18f2ca81 : 00007ff7`85c82a78 00000000`00040440
00000000`00000000 00007ff7`85c82910 : sqllang!SQLDumperLibraryInvoke+0x1cd
00000052`b9e7b350 00007ffb`18f2d534 : 00000000`00025160 000001da`ae1bd2d0
00000000`00000000 00000000`00000000 : sqllang!SQLLangDumperLibraryInvoke+0x171
00000052`b9e7b410 00007ffb`18ef6cce : 00000052`b9e7eea0 000001da`ae1bd2d0
00000052`b9e7eea0 000001da`ae1bd2d0 : sqllang!CImageHelper::DoMiniDump+0x4a5
00000052`b9e7b620 00007ffb`18285ae9 : 00000000`00000000 00000052`b9e7d100
00000000`000000ff 00000000`0000083f : sqllang!stackTrace+0x99e
00000052`b9e7d050 00007ffb`1716cfb2 : 00000000`00000008 00000000`00000034
000001e3`dff44160 000001d6`f21b26f0 : sqllang!stackTraceCallBack+0x99
00000052`b9e7d0a0 00007ffb`1712e1e8 : 000001e3`dff44160 00000000`d57e18d0
000001e4`ffffffff 000001e4`f3080048 : sqldk!ex_dump_if_requested+0x142
00000052`b9e7d340 00007ffb`1712e04a : 00000000`00000034 00000000`0000002b
00000000`00000016 00007ffb`00000008 : sqldk!ex_raise2+0x2de
00000052`b9e7d670 00007ffb`1bfd67ed : 00000001`63190430 00000001`1a6c0001
00000005`00193c8b 00007ffb`00000008 : sqldk!ex_raise+0xc4
00000052`b9e7d6f0 00007ffb`1b64bdac : 00000000`00000000 00000052`00000008
000145d7`25cfc8e0 00000052`b9e7daf0 : sqlmin!RaiseInconsistencyError+0x5ed
00000052`b9e7da40 00007ffb`1b86fc53 : 00000000`00000001 00000052`b9e7dba0
00000000`00000000 00000000`ffffffff : sqlmin!IAMPageRef::Fix+0x19a
00000052`b9e7daa0 00007ffb`1b86fdcd : 000001e4`ffffffff 00000001`00000001
000001e0`259881e0 000001e3`dff45420 : sqlmin!DeferredAllocUnitDrop::DeAllocateFirstIAMAndItsExtents+0x474
00000052`b9e7dc20 00007ffb`1b870058 : 000001e4`f3362520 000001e4`e502dee0
000001e3`dff44160 000001e4`e502dee0 : sqlmin!DeferredAllocUnitDrop::ProcessOneIAM+0x8e
00000052`b9e7dce0 00007ffb`1b86f296 : 000001e4`e502dee0 000001e4`f3362520
000001e3`dff45460 00000000`00000520 : sqlmin!DeferredAllocUnitDrop::Process+0x178
00000052`b9e7ddc0 00007ffb`1a6d882a : 00000000`0000ffff 00007ffb`1cb6d2e0
000001da`ae1bc810 00007ffb`1a6c52a0 : sqlmin!DeferredAllocUnitDrop::AllocUnitCleanupTask+0x77
00000052`b9e7de00 00007ffb`1a6d8499 : 0000aa74`003f79f4 00000000`00000000
00000052`b9e7deac 00000052`b9e7df88 : sqlmin!PeriodicTimerTask::ProcessTskPkt+0xa
00000052`b9e7de30 00007ffb`1a6d8375 : 00007ffb`1cb6d2e0 000001e3`00000000
000001da`00000000 000001da`ae1bc830 : sqlmin!TaskReqPktTimer::ExecuteTask+0x69
00000052`b9e7df20 00007ffb`17f4492a : 000001e4`e3877500 00000000`00000000
00000000`00000001 00000000`00000001 : sqlmin!OnDemandTaskContext::ProcessTskPkt+0x31f
00000052`b9e7e110 00007ffb`1a6d80a4 : ffefffff`ffffffff 00000000`00000000
000001e4`e6612160 00007ffb`17113ae7 : sqllang!SystemTaskEntryPoint+0x3d2
00000052`b9e7ef80 00007ffb`17114e4d : 00000000`00000000 00000000`00000011
00000000`00000000 000001e4`f3180040 : sqlmin!OnDemandTaskContext::FuncEntryPoint+0x25
00000052`b9e7efb0 00007ffb`171153b8 : 000001da`8b07a908 000001da`8b07a8c8
000001da`8b07a950 000001da`00000000 : sqldk!SOS_Task::Param::Execute+0x231
00000052`b9e7f5b0 00007ffb`171151ad : 000001da`8b07a8d8 000001e4`f3180040
000001e4`f3180040 000001e3`dff44160 : sqldk!SOS_Scheduler::RunTask+0xad
00000052`b9e7f620 00007ffb`17136758 : 00000000`00000000 000001e3`dff44160
000001e3`dff44160 00000000`00000000 : sqldk!SOS_Scheduler::ProcessTasks+0x3cd
00000052`b9e7f710 00007ffb`17136850 : 000001e3`dff44160 00000000`00000000
0000011d`38ac788b 0000011d`38ac788b : sqldk!SchedulerManager::WorkerEntryPoint+0x2a1
00000052`b9e7f7e0 00007ffb`17135d37 : 000001d6`f2193d08 00000052`b9e7f870
000001e4`f3080270 000001e4`f3080270 : sqldk!SystemThread::RunWorker+0x8f
00000052`b9e7f810 00007ffb`17136289 : 000001e4`f3080230 00000000`00000000
00000000`00000000 000001e4`f3080170 : sqldk!SystemThreadDispatcher::ProcessWorker+0x2e7
00000052`b9e7f8b0 00007ffb`339684d4 : 00000000`00000000 00000000`00000000
000001d6`f2193d08 000001d6`f2193d08 : sqldk!SchedulerManager::ThreadEntryPoint+0x1d8
00000052`b9e7f960 00007ffb`33d3e851 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
00000052`b9e7f990 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21
THREAD_SHA1_HASH_MOD_FUNC: 88f37641914f7d930e4fdf160ea8da0a54ef5b81
THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 9ad82aab5350c2af8fcac10397d276767dfdeb16
THREAD_SHA1_HASH_MOD: 61835a57e65e7bf7271c557b06634d49bb69a250
FOLLOWUP_IP:
sqllang!SQLDumperLibraryInvoke+1cd
00007ffb`182859ad 448bf0 mov r14d,eax
FAULT_INSTR_CODE: 45f08b44
SYMBOL_STACK_INDEX: 2
SYMBOL_NAME: sqllang!SQLDumperLibraryInvoke+1cd
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: sqllang
IMAGE_NAME: sqllang.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 5cdceefa
STACK_COMMAND: ~0s ; .ecxr ; kb
BUCKET_ID: APPLICATION_FAULT_sqllang!SQLDumperLibraryInvoke+1cd
FAILURE_EXCEPTION_CODE: 42ac
FAILURE_IMAGE_NAME: sqllang.dll
BUCKET_ID_IMAGE_STR: sqllang.dll
FAILURE_MODULE_NAME: sqllang
BUCKET_ID_MODULE_STR: sqllang
FAILURE_FUNCTION_NAME: SQLDumperLibraryInvoke
BUCKET_ID_FUNCTION_STR: SQLDumperLibraryInvoke
BUCKET_ID_OFFSET: 1cd
BUCKET_ID_MODTIMEDATESTAMP: 5cdceefa
BUCKET_ID_MODCHECKSUM: 275f227
BUCKET_ID_MODVER_STR: 2017.140.3162.1
BUCKET_ID_PREFIX_STR: APPLICATION_FAULT_
FAILURE_PROBLEM_CLASS: APPLICATION_FAULT
FAILURE_SYMBOL_NAME: sqllang.dll!SQLDumperLibraryInvoke
FAILURE_BUCKET_ID: APPLICATION_FAULT_42ac_sqllang.dll!SQLDumperLibraryInvoke
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/sqlservr.exe/2017.140.3162.1/5cdced6b/sqllang.dll/2017.140.3162.1/5cdceefa/42ac/003459ad.htm?Retriage=1
TARGET_TIME: 2019-06-24T03:04:00.000Z
OSBUILD: 9200
OSSERVICEPACK: 2969
SERVICEPACK_NUMBER: 0
OS_REVISION: 0
SUITE_MASK: 400
PRODUCT_TYPE: 3
OSPLATFORM_TYPE: x64
OSNAME: Windows 8
OSEDITION: Windows 8 Server TerminalServer DataCenter SingleUserTS
USER_LCID: 0
OSBUILD_TIMESTAMP: 2019-05-04 00:40:41
BUILDDATESTAMP_STR: 190503-1820
BUILDLAB_STR: rs1_release
BUILDOSVER_STR: 10.0.14393.2969
ANALYSIS_SESSION_ELAPSED_TIME: 1ac1
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING:
um:application_fault_42ac_sqllang.dll!sqldumperlibraryinvoke
FAILURE_ID_HASH: {ab1e43e2-da46-c388-f696-a374d74f9a5d}
Followup: MachineOwner
---------
Cause.
Finding "sqlmin!RaiseInconsistencyError+0x5ed" and "sqlmin!IAMPageRef::Fix+0x19a"
led me to believe the database has physical or logical inconsistencies and that
the database needed a repair. The customer only performs full backups every day.
Later on, the customer reported the following error trying to create an
index.
Msg 5243, Level 22, State 8, Line 6
An inconsistency was detected during an internal operation. Please contact
technical support.
Solution.
I ran DBCC CHECKDB with REPAIR_ALLOW_DATA_LOSS as shown below.
ALTER DATABASE [DatabaseName] SET EMERGENCY;
GO
DBCC CHECKDB (N'DatabaseName', REPAIR_ALLOW_DATA_LOSS) WITH ALL_ERRORMSGS,
NO_INFOMSGS;
GO
Allocation errors and inconsistency errors were found and fixed by DBCC
CHECKDB.
![[Company Logo Image]](images/SQLCofee.jpg)
