#dev-alerts has a problem. It gets 40–60 automated messages per day. Two or three
of them actually require human action. The other 57 are noise — and every engineer
on the team knows it. When the channel is 95% noise, the 5% signal gets ignored
too. You've built an alert system that produces the opposite of alertness.
The Slack alert graveyard
Most engineering teams start with good intentions. They wire up GitHub CI
notifications to Slack so the team knows when builds fail. Then they add
deployment status alerts. Then error monitoring. Then uptime monitoring.
Then Dependabot security alerts.
Each addition is individually justified. Together, they create a channel that
nobody watches with attention because watching it with attention would require
processing 60 messages per day, the majority of which require no action.
The research on alert fatigue in engineering teams is unambiguous:
- The average engineering Slack workspace has 4–7 channels receiving automated engineering notifications
- Fewer than 20% of automated notifications receive any human response within 2 hours
- Engineers who work in teams with high alert volumes are 3× more likely to report "notification blindness" — the habit of mentally filtering Slack as background noise
- Teams with notification blindness have significantly higher mean time to respond to real incidents — the alerts that need a response don't get one faster than teams with no alerts at all
The worst outcome isn't that your team ignores noise. It's that they learn to
ignore the signal too — because the signal looks exactly like the noise.
The anatomy of a Slack alert that gets acted on
The difference between a notification that gets acted on and one that gets
ignored isn't loudness or volume — it's context. A notification that contains
enough information to make an immediate decision gets attention. A notification
that requires a 5-minute investigation before you even know whether you need
to care gets skipped.
A Slack alert that gets acted on has five components:
- 1. Who needs to act. The alert is directed — either a specific @mention, a specific channel with a specific audience, or both. "CI failed" sent to #dev-general means nobody in particular is responsible. "CI failed on your PR [link]" sent to @engineer means one person is clearly responsible.
- 2. What specifically happened. Not "build failed" — "Build failed: Payment webhook integration tests (3 failures). Commit abc123 by @engineer, merged 14 minutes ago." The specific failing suite and the responsible commit are actionable. "Build failed" is a starting point for a 10-minute investigation.
- 3. What the pre-existing state was. Context about whether this is a new failure or a recurring one. "This test suite has failed 3 times in the last 7 days" changes the priority calculus. "This is the first failure in 30 days" changes it differently.
- 4. What the recommended action is. Not prescriptive — but a structured starting point. "Review build log [link] or check recent commits touching PaymentWebhookHandler." The recommended action reduces the time from "I see this" to "I'm working on it."
- 5. A single clear link. One link to the most relevant destination — the Jira ticket created from this failure, the build log, or the PR. Not three links with different levels of relevance. One link, the best one.
The five alert categories and which ones should never go to Slack
Not all engineering events belong in Slack. The mistake most teams make is
treating Slack as the universal notification endpoint — if something can trigger
an alert, it goes to Slack.
Here's a framework for categorizing alerts by appropriate destination:
- Category 1: Requires immediate human action (production incidents, main branch CI failures). → Send to Slack #on-call with @mention of responsible engineer. These are the alerts that need fast human response. Keep this category narrow — the more events are in this category, the less urgent it feels.
- Category 2: Requires human action within the workday (preview deployment failures, PR review escalations). → Send as direct message to the responsible engineer, not to a channel. Channel messages for work-hours issues train the channel to feel like low-priority work. DMs feel more like a direct request.
- Category 3: Requires human action within the sprint (stale PRs, flaky test accumulation, TODO debt threshold crossed). → Create a ticket and send a weekly digest, not individual Slack messages. Stale PR notifications should fire as a batch summary ("5 PRs are over 48h old"), not as individual messages for each one.
- Category 4: Informational, no action required (successful deployments, PR merged, CI green after fixing). → Do not send to Slack. These are the noise messages that train engineers to ignore the channel. If you want a deployment history, put it in a searchable log — not a real-time Slack message.
- Category 5: Security and compliance (Dependabot alerts, secret scanning, permission changes). → Route to a dedicated #security-alerts channel watched by the security owner, not to #dev-alerts. Mixing security alerts into the general engineering notification stream guarantees they get ignored.
Building the escalation ladder: Slack first, ticket second, page third
The escalation ladder defines what happens when a Slack alert doesn't get a response.
Most teams have an implicit ladder (someone eventually notices); few have an
explicit one that's wired into the tooling.
The pattern that works:
- T+0 minutes: Slack notification fires. Directed to the responsible engineer with full context. For P1 events, also sent to #on-call.
- T+15 minutes: If no response, ticket is escalated. The automatically created Jira/Linear ticket is updated to reflect that it's been unacknowledged for 15 minutes and the severity is increased. This is an automatic state change, not a human decision.
- T+30 minutes: If P1 still unacknowledged, page the on-call engineer. Slack is not a reliable paging mechanism. For genuinely urgent production issues that haven't been picked up, a PagerDuty or direct escalation path should fire. Don't rely on the engineer noticing a Slack message for production-down scenarios.
The escalation ladder makes the implicit explicit: it removes the ambiguity
of "did someone see this?" and replaces it with a deterministic process.
Engineers can trust that unresponded alerts will escalate — so they don't
need to manually monitor channels to catch things that might slip through.
The Slack engineering alert configuration checklist
Channel setup
- Separate channels for different urgency levels: #incidents (P1/P2), #dev-alerts (P3/info), #security-alerts (security only)
- #incidents channel has limited membership — only on-call engineers and EMs, not the whole engineering team
- #dev-alerts is opt-in, not mandatory — engineers working on unrelated services shouldn't be in every alert channel
Notification content rules
- Every automated alert includes: what happened, specific context (commit/PR/engineer), recommended action, single best link
- Successful events (deploys succeeded, CI green) do NOT go to Slack — they go to a searchable log
- Batch weekly digests for Category 3 events (stale PRs, flaky tests, tech debt) — not individual per-event messages
Routing logic
- P1 events: channel alert + @responsible engineer DM
- P2-P3 events: DM only to responsible engineer
- Security events: #security-alerts only, not #dev-alerts
- All alerts also create a structured ticket — Slack is triage, tickets are accountability
Escalation rules
- P1 unacknowledged for 15 minutes → escalate ticket severity, resend to #incidents
- P1 unacknowledged for 30 minutes → trigger on-call page (not just Slack)
- All escalation timing is automated — not dependent on a human noticing the gap
The teams with the lowest alert fatigue are not the ones that send fewer alerts.
They're the ones that send better alerts — specific, contextualized,
directed, and escalating. When every Slack notification has clear context and
a clear owner, engineers stop treating the channel as noise. They treat it as
a work queue.
The difference between a #dev-alerts graveyard and an effective alert channel
is almost entirely in the context layer — and building that context layer
automatically, from your CI and deployment events, is exactly the problem
an automation engine is designed to solve.
Wire your CI failures and deployment events to structured Slack alerts. Connect Slack in Deviera.