Part 3 — Log, Trend, and Dashboard Your SQL Error and Warning Findings

This builds on Part 2. Instead of only emailing what happened in the last 24 hours, we will:

  • Store a summary of errors and warnings in a small table each run
  • Query that table for trends by day and week
  • (Optionally) ship those summaries to a central ops database from many servers

Everything below is copy-paste ready. Replace placeholders like YourUtilityDB, YourDatabaseMailProfile, ops@example.com, and linked server names as needed.

USE [YourUtilityDB];
GO

IF OBJECT_ID('dbo.ErrorLogSummary','U') IS NULL
BEGIN
    CREATE TABLE dbo.ErrorLogSummary
    (
        SummaryDate      date           NOT NULL,
        InstanceName     sysname        NOT NULL,
        Source           nvarchar(20)   NOT NULL,   -- 'SQL' or 'Agent'
        SeverityClass    varchar(10)    NOT NULL,   -- 'ERROR' or 'WARNING'
        Tag              nvarchar(50)   NOT NULL,   -- friendly label, e.g. 'Autogrow'
        NormalizedText   nvarchar(400)  NOT NULL,   -- trimmed message for grouping
        Occurrences      int            NOT NULL,
        FirstSeen        datetime       NOT NULL,
        LastSeen         datetime       NOT NULL,
        CONSTRAINT PK_ErrorLogSummary 
            PRIMARY KEY CLUSTERED 
            (SummaryDate, InstanceName, Source, SeverityClass, Tag, NormalizedText)
    );

    CREATE INDEX IX_ErrorLogSummary_LastSeen 
        ON dbo.ErrorLogSummary(LastSeen) INCLUDE (SeverityClass, Tag, Occurrences);
END
GO
B. Stored proc to capture and store findings

This is a compact version of Part 2 that reads both SQL Server and SQL Agent logs, classifies messages with the same rules, groups identical messages, and upserts into dbo.ErrorLogSummary.
USE [YourUtilityDB];
GO

IF OBJECT_ID('dbo.Capture_ErrorLogFindings','P') IS NOT NULL
    DROP PROCEDURE dbo.Capture_ErrorLogFindings;
GO

CREATE PROCEDURE dbo.Capture_ErrorLogFindings
    @HoursBack int = 24
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE 
        @Now    datetime2(0) = SYSDATETIME(),
        @Start  datetime2(0) = DATEADD(HOUR, -@HoursBack, SYSDATETIME()),
        @Inst   sysname      = CAST(SERVERPROPERTY('ServerName') AS sysname),
        @SDate  date         = CONVERT(date, SYSDATETIME());

    IF OBJECT_ID('tempdb..#Log') IS NOT NULL DROP TABLE #Log;
    CREATE TABLE #Log
    (
        Source       nvarchar(20),   -- 'SQL' or 'Agent'
        LogDate      datetime,
        ProcessInfo  nvarchar(50),
        [Text]       nvarchar(max)
    );

    -- SQL Server error log
    INSERT INTO #Log (Source, LogDate, ProcessInfo, [Text])
    EXEC master.dbo.xp_readerrorlog 0, 1, NULL, NULL, @Start, @Now, N'desc';
    UPDATE #Log SET Source = 'SQL' WHERE Source IS NULL;

    -- SQL Agent log
    INSERT INTO #Log (Source, LogDate, ProcessInfo, [Text])
    EXEC master.dbo.xp_readerrorlog 0, 2, NULL, NULL, @Start, @Now, N'desc';
    UPDATE #Log SET Source = 'Agent' WHERE Source IS NULL AND ProcessInfo IS NOT NULL;

    -- Rules
    DECLARE @Rules TABLE (Pattern nvarchar(200), Class varchar(10), Weight int, Tag nvarchar(50));
    INSERT INTO @Rules VALUES
    (N'%Error: 823%', 'ERROR', 2, N'I/O'),
    (N'%Error: 824%', 'ERROR', 2, N'I/O'),
    (N'%Error: 825%', 'ERROR', 2, N'I/O-Retry'),
    (N'%Error: 9002%', 'ERROR', 2, N'LogFull'),
    (N'%Error: 1101%', 'ERROR', 2, N'FileFull'),
    (N'%Error: 1105%', 'ERROR', 2, N'FileFull'),
    (N'%Error: 3624%', 'ERROR', 2, N'Assertion'),
    (N'%Error: 3041%', 'ERROR', 2, N'BackupFail'),
    (N'%backup%failed%', 'ERROR', 2, N'BackupFail'),
    (N'%assertion%', 'ERROR', 2, N'Assertion'),
    (N'%stack dump%', 'ERROR', 2, N'Dump'),
    (N'%is marked SUSPECT%', 'ERROR', 2, N'DBState'),
    (N'%failed to start%', 'ERROR', 2, N'Service'),
    (N'%Autogrow of file%', 'WARNING', 1, N'Autogrow'),
    (N'%Login failed for user%', 'WARNING', 1, N'LoginFailed'),
    (N'%SSPI handshake%', 'WARNING', 1, N'Kerberos'),
    (N'%I/O requests taking longer%', 'WARNING', 1, N'LongIO'),
    (N'%availability group%', 'WARNING', 1, N'AG'),
    (N'%availability replica%', 'WARNING', 1, N'AG'),
    (N'%redo queue%', 'WARNING', 1, N'AG'),
    (N'Warning:%', 'WARNING', 1, N'Generic');

    -- Noise list
    DECLARE @Noise TABLE (Pattern nvarchar(200));
    INSERT INTO @Noise VALUES
    (N'%Using%backup%compression%'),
    (N'%registry%is%configured%for%memory%'),
    (N'%The%Service%Broker%'),
    (N'%Database%option%SET%'),
    (N'%successfully%opened%'),
    (N'%Login%successful%'),
    (N'%Detected%aligned%IO%'),
    (N'%Error log has been reinitialized%');

    ;WITH Base AS
    (
        SELECT *
        FROM #Log
        WHERE [Text] IS NOT NULL
          AND NOT EXISTS (SELECT 1 FROM @Noise n WHERE #Log.[Text] LIKE n.Pattern)
    ),
    Tagged AS
    (
        SELECT b.*,
               r.Class,
               r.Weight,
               r.Tag
        FROM Base b
        OUTER APPLY
        (
            SELECT TOP (1) *
            FROM @Rules r
            WHERE b.[Text] LIKE r.Pattern
            ORDER BY r.Weight DESC
        ) r
    ),
    Classified AS
    (
        SELECT 
            b.Source,
            b.LogDate,
            b.ProcessInfo,
            b.[Text],
            COALESCE(r.Class,'INFO') AS SeverityClass,
            COALESCE(r.Tag,  'General') AS Tag
        FROM Tagged b
        LEFT JOIN @Rules r
            ON 1 = 0 -- r already applied in OUTER APPLY
    )
    SELECT *
    INTO #Findings
    FROM Classified
    WHERE SeverityClass IN ('ERROR','WARNING');

    ALTER TABLE #Findings ADD NormalizedText AS LEFT([Text], 400);

    SELECT 
        @SDate                 AS SummaryDate,
        @Inst                  AS InstanceName,
        Source,
        SeverityClass,
        Tag,
        NormalizedText,
        COUNT(*)               AS Occurrences,
        MIN(LogDate)           AS FirstSeen,
        MAX(LogDate)           AS LastSeen
    INTO #Groups
    FROM #Findings
    GROUP BY Source, SeverityClass, Tag, NormalizedText;

    -- Upsert into summary table
    MERGE dbo.ErrorLogSummary AS T
    USING #Groups AS S
      ON  T.SummaryDate    = S.SummaryDate
      AND T.InstanceName   = S.InstanceName
      AND T.Source         = S.Source
      AND T.SeverityClass  = S.SeverityClass
      AND T.Tag            = S.Tag
      AND T.NormalizedText = S.NormalizedText
    WHEN MATCHED THEN
        UPDATE SET 
            T.Occurrences = T.Occurrences + S.Occurrences,
            T.FirstSeen   = CASE WHEN S.FirstSeen < T.FirstSeen THEN S.FirstSeen ELSE T.FirstSeen END,
            T.LastSeen    = CASE WHEN S.LastSeen  > T.LastSeen  THEN S.LastSeen  ELSE T.LastSeen  END
    WHEN NOT MATCHED THEN
        INSERT (SummaryDate, InstanceName, Source, SeverityClass, Tag, NormalizedText, Occurrences, FirstSeen, LastSeen)
        VALUES (S.SummaryDate, S.InstanceName, S.Source, S.SeverityClass, S.Tag, S.NormalizedText, S.Occurrences, S.FirstSeen, S.LastSeen);

END
GO

Schedule it

Create a SQL Agent job that runs:

EXEC YourUtilityDB.dbo.Capture_ErrorLogFindings @HoursBack = 24;

Run it once daily. On critical systems, run it every hour and still summarize by SummaryDate.


C. Quick “dashboard” queries

Paste these in a SQL notebook or SSMS. They give you an at-a-glance view.

Top recurring warnings in the last 7 days

SELECT TOP (20)
    Tag,
    NormalizedText,
    SUM(Occurrences) AS TotalHits
FROM dbo.ErrorLogSummary
WHERE SummaryDate >= DATEADD(DAY, -7, CONVERT(date, GETDATE()))
  AND SeverityClass = 'WARNING'
GROUP BY Tag, NormalizedText
ORDER BY TotalHits DESC;

All errors in the last 7 days, most recent first

SELECT 
    SummaryDate, Source, Tag, Occurrences, LastSeen, NormalizedText
FROM dbo.ErrorLogSummary
WHERE SummaryDate >= DATEADD(DAY, -7, CONVERT(date, GETDATE()))
  AND SeverityClass = 'ERROR'
ORDER BY LastSeen DESC;

Weekly trend by severity

WITH W AS
(
    SELECT 
        DATEFROMPARTS(YEAR(SummaryDate), 1, 1) 
          + (DATEPART(ISO_WEEK, SummaryDate)-1)*7 AS WeekStart,
        SeverityClass,
        SUM(Occurrences) AS Hits
    FROM dbo.ErrorLogSummary
    GROUP BY DATEPART(ISO_WEEK, SummaryDate), YEAR(SummaryDate), SeverityClass, 
             DATEFROMPARTS(YEAR(SummaryDate), 1, 1) + (DATEPART(ISO_WEEK, SummaryDate)-1)*7
)
SELECT WeekStart, SeverityClass, SUM(Hits) AS Hits
FROM W
GROUP BY WeekStart, SeverityClass
ORDER BY WeekStart DESC, SeverityClass;

Which instances are the noisiest this month

SELECT
InstanceName,
SeverityClass,
SUM(Occurrences) AS TotalHits
FROM dbo.ErrorLogSummary
WHERE SummaryDate >= DATEFROMPARTS(YEAR(GETDATE()), MONTH(GETDATE()), 1)
GROUP BY InstanceName, SeverityClass
ORDER BY TotalHits DESC;

D. Optional: centralize across many servers

You have two easy patterns.

Option 1: Linked Server push

  1. On the central server, create the same table:
-- On CENTRAL server
USE [OpsDB];
-- create dbo.ErrorLogSummary with the same schema as Section A

2. On each source server, create a linked server named LS_CENTRAL that points to the central instance.

3. Add a second Agent job step after Capture_ErrorLogFindings to push today’s rows:

DECLARE @Today date = CONVERT(date, GETDATE());

;WITH Src AS
(
    SELECT *
    FROM YourUtilityDB.dbo.ErrorLogSummary
    WHERE SummaryDate = @Today
)
MERGE [LS_CENTRAL]...[OpsDB].dbo.ErrorLogSummary AS T
USING Src AS S
  ON  T.SummaryDate    = S.SummaryDate
  AND T.InstanceName   = S.InstanceName
  AND T.Source         = S.Source
  AND T.SeverityClass  = S.SeverityClass
  AND T.Tag            = S.Tag
  AND T.NormalizedText = S.NormalizedText
WHEN MATCHED THEN
  UPDATE SET 
    T.Occurrences = T.Occurrences + S.Occurrences,
    T.FirstSeen   = CASE WHEN S.FirstSeen < T.FirstSeen THEN S.FirstSeen ELSE T.FirstSeen END,
    T.LastSeen    = CASE WHEN S.LastSeen  > T.LastSeen  THEN S.LastSeen  ELSE T.LastSeen  END
WHEN NOT MATCHED THEN
  INSERT (SummaryDate, InstanceName, Source, SeverityClass, Tag, NormalizedText, Occurrences, FirstSeen, LastSeen)
  VALUES (S.SummaryDate, S.InstanceName, S.Source, S.SeverityClass, S.Tag, S.NormalizedText, S.Occurrences, S.FirstSeen, S.LastSeen);

Option 2: Pull from central

Run a central job that loops through linked servers and pulls today from each with OPENQUERY. Use whichever your security model prefers.


E. Reporting ideas

  • SSRS or Power BI: point to the central table and add visuals for:
    • Errors by day and tag
    • Warnings by week
    • “Top offenders” messages
    • Instances with the most activity
  • Email digest: add a small proc on the central server that emails a weekly rollup using the same HTML table pattern you liked in Part 2.

F. Maintenance tips

  • Keep sp_cycle_errorlog and sp_cycle_agent_errorlog on a weekly schedule so raw logs stay readable.
  • Trim dbo.ErrorLogSummary to a rolling window. For example, keep 180 days:
DELETE FROM dbo.ErrorLogSummary
WHERE SummaryDate < DATEADD(DAY, -180, CONVERT(date, GETDATE()));
  • If a single message grows beyond 400 characters, it will be trimmed. For full-text needs, store raw rows in a separate history table, but that is usually not necessary for trends.

Series Summary: Error Log Reviews → Automation → Trends

Part 1 — Review the SQL Error Log like a DBA

  • Where: SSMS → Management → SQL Server Logs → Current (and SQL Agent → Error Logs).
  • What to scan daily: 823/824/825 (I/O), 9002 (log full), 1101/1105 (file full), 3624 (assertion), 3041/backup failed, suspect DB, unplanned restarts/dumps, Always On disconnects.
  • High-signal warnings: frequent autogrows, spikes in 18456 login failures, SSPI handshake failures, long I/O warnings, AG redo/backlog.
  • Quick filters: use xp_readerrorlog with simple text filters (I/O, Login failed, 3041).
  • Hygiene: keep several rollover files; run sp_cycle_errorlog and sp_cycle_agent_errorlog on a schedule.

Part 2 — Automate and Email a Daily Summary

  • What the job does: scans the last N hours of SQL and SQL Agent logs, classifies Errors vs Warnings, suppresses known noise, groups duplicates with counts, and emails an HTML summary.
  • Why it helps: cuts noise, highlights patterns (e.g., 50 autogrows vs 1), and adds Agent job failures to one report.
  • Operate it: run via SQL Agent (daily for most servers; 15–60 min on critical). Email only when findings exist or always send a digest.
  • Tune points: adjust the time window, add/remove patterns, and extend the noise list for your environment.

Part 3 — Store, Trend, and Centralize

  • Storage: write grouped findings to a small table (ErrorLogSummary) each run.
  • Trends: query by day/week to see which warnings/errors repeat, which instances are noisiest, and how patterns change over time.
  • Dashboards: use the included queries for “top warnings,” “recent errors,” weekly severity trends, and per-instance volume.
  • Fleet view: push or pull summaries to a central ops database via linked servers and report from there.
  • Retention: keep ~180 days, trim older rows.

Recommended cadence and thresholds

  • Cadence:
    • Production: review daily (automated email) + on-demand for incidents.
    • Mission-critical: automated every 15–60 minutes; daily digest for record.
  • Treat as “investigate now”: any 823/824/825, 9002, 1101/1105, 3624, 3041/backup failed, suspect DB, dumps, unplanned restart, repeated long I/O, sustained AG sync problems.
  • Treat as “watch and fix soon”: repeated autogrows, rising login failures from a single host, recurring SSPI warnings, frequent Agent job failures.

Bottom line: Part 1 teaches the manual review, Part 2 makes it automatic and readable, and Part 3 turns the signal into trends and a fleet-level view.


Discover more from SQLYARD

Subscribe to get the latest posts sent to your email.

Leave a Reply

Discover more from SQLYARD

Subscribe now to keep reading and get access to the full archive.

Continue reading