Scope the deploy downtime before the 22:00 migration

Scope a downtime instead of muting monitors

The companion lists the monitors whose scope covers the service you are about to touch, names the four that will fire, and proposes the downtime with its window and tag scope written out. You approve it, it schedules it, and it posts the event so tomorrow morning nobody wonders what that dip was.

Then comes the listing nobody runs by hand: every scheduled suppression with its scope and id. Cancelling one resumes alerting immediately. On a Monday routine that is three lines back instead of a dashboard nobody has bookmarked, which is operations work that stops needing a person to hold it.

What can a companion do in Datadog?

Monitors and downtimes, which is the alerting layer, plus hosts, dashboards to read, metric timeseries queries and a posted event. That covers the deploy window, the morning sweep of what fired, and the annotation everybody sees on tomorrow’s graphs.

Worth knowing while you plan: there is no log search tool, and nothing for APM traces, SLOs, Synthetics, RUM, Incident Management or notebooks. Dashboards are read-only, so a companion describes what one contains, and metrics come through the timeseries endpoint with a from and to in Unix seconds.

• Monitors: list, get, create, update, delete, mute, unmute.
• Downtimes: list, schedule, cancel. The primitive to reach for around a deploy.
• Dashboards: list and get only. Nothing creates or edits one.
• Metrics and hosts: list metrics reporting in a window, run a timeseries query, list hosts.
• Events: post a single event to the Datadog event stream.

Should a companion be creating monitors?

For most teams, yes.

A monitor needs a type, a query and a name, and the options carry the thresholds and notification targets, which is the part people get wrong writing it at 2 a.m. Updating changes only the fields supplied, which is the safe behaviour, and the approval card names what moves.

Deleting is where to be deliberate. The delete tool accepts a force flag that bypasses Datadog’s own reference checks for SLOs and composite monitors pointing at the one being removed. The approval gate is real, and deciding in advance which key a companion holds is the other half of it.

Get the overnight write-up before standup

A monitor firing is a fact.

Why it fired is almost always somewhere else: a deploy, a config change, a vendor status page, a ticket somebody filed forty minutes earlier. A companion reads the monitor, then reads those in the accounts you are signed into, and writes the paragraph that joins them.

That write-up is the deliverable. What fired overnight, which of those were one alert three times, which host stopped reporting, delivered before standup. Assembling a week of it into a table is data extraction with a metric query attached to each row.

Frequently asked questions