AI Marketing System Health Check

Healthy is not the same as running

A live AI marketing system can fail in ways that produce no error at all. A scheduled job keeps running but processes nothing. A tracking change doubles reported conversions and automated bidding follows the false signal. Drafts slowly drift from the brand voice because the context document is six months out of date. None of these shows up as a red light. They show up, eventually, as results that are worse than the reports suggest.

This health check is for systems that already exist. If you have not built anything yet, the readiness checklist is the better starting point.

Assessment

AI marketing system health check

Answer for the system as it runs today. Yes = 2, partly = 1, no = 0. Score 2 for yes, 1 for partly, 0 for no.

1. 1. For each workflow, can you say how many items it should have produced last week and confirm that it did?

   A job that runs but produces nothing looks identical to one with nothing to do unless you check the expected count.

   YesPartlyNo
2. 2. Do items the system cannot handle go to a named person, and is someone watching how many there are?

   A rising exception rate is often the first sign that an input or dependency has changed.

   YesPartlyNo
3. 3. Has the context the system uses (offer, prices, proof, tone) been updated in the last three months?

   Stale context produces confident, out-of-date output.

   YesPartlyNo
4. 4. Is every customer-facing output checked by a person before it goes out, or covered by a rule you have deliberately agreed?

   Unreviewed output is where brand, legal and accuracy risk accumulates.

   YesPartlyNo
5. 5. Do you sample finished outputs each month and score them against a written standard?

   Gradual quality drift is invisible without a fixed yardstick.

   YesPartlyNo
6. 6. Do tracked conversions still reconcile with your own orders or CRM records within a difference you can explain?

   Automated optimisation follows whatever is tracked, right or wrong.

   YesPartlyNo
7. 7. Can you show the outcome each workflow was built to move, against its original baseline?

   Without it, the system may be busy rather than useful.

   YesPartlyNo
8. 8. Is there a log of what each workflow did, what failed and what changed?

   When something goes wrong, the log is the difference between a fix in minutes and a guess.

   YesPartlyNo
9. 9. Are actions that spend money, publish or contact customers limited by approvals or caps that you have tested recently?

   A control that has never been tested may not work when it is needed.

   YesPartlyNo
10. 10. Do you know the monthly running cost of each workflow and roughly what each output costs?

    Costs from usage-based tools can creep up without anyone deciding to spend more.

    YesPartlyNo

Answered 0 of 10. Your result appears when every question is answered.

What each score range means

• 0 to 7: At risk. The system may be failing silently, optimising towards unreliable numbers, or sending unchecked work. Reports cannot be taken at face value. Next step: Pause any automated spend or sending that depends on questions 6 or 9, then fix output counting, reconciliation and approvals before anything else.
• 8 to 14: Working but fragile. The system does useful work, but at least one area would let a failure go unnoticed for weeks. Next step: Add expected-output counts and an exception watch, refresh the context document, and start a monthly sample review.
• 15 to 20: Healthy. Failures would be noticed quickly, numbers are trustworthy and people control what leaves the business. Next step: Repeat quarterly and after any tracking or tool change; consider extending to the next workflow.

What your band means

0 to 7, at risk. Do not add anything new. The priority is to make the existing system trustworthy: confirm it produces what it should, confirm the numbers it optimises towards are real, and confirm nothing leaves the business unchecked. This often takes a week or two and can reveal that recent "wins" were measurement artefacts.

8 to 14, working but fragile. The most common result. The system is useful but depends on someone happening to notice problems. The fixes are small and mechanical: counts, logs, a sample review. They are dull, and they are what makes the difference between a system and a set of scripts.

15 to 20, healthy. Keep the rhythm. A healthy score decays quietly when tools update, staff change or the offer moves on, so put the next check in the calendar now.

The four failure types, and how to catch each

Silent failure

The system runs and produces nothing, or far less than it should. The check is to compare what each workflow produced against what it should have produced given its inputs. Exit messages and "last run" timestamps are not evidence of work done; counts of outputs are.

Drift

Output quality declines slowly. Prompts are edited, tools update their models, context goes stale. Take a monthly sample of ten outputs and score them against a written standard, the same standard every month, so a decline is visible as a number rather than a feeling.

False signal

The system optimises towards a number that has stopped meaning what you think. A tag fires twice, a form changes, a consent banner update changes what is observed. Google Ads help describes consent mode modelling, under which conversions from users who decline consent can be modelled and shown alongside observed ones in reports. Check your own account's conversion settings to see whether that applies to you. Where it does, it is legitimate, but platform totals are then partly estimates, and they should be reconciled with your own records regularly.

Broken guardrail

A cap, approval or rule that exists on paper but no longer works. The experimentation literature treats this seriously: Kohavi, Tang and Xu's Trustworthy Online Controlled Experiments discusses sample ratio mismatch and other trust-related guardrail metrics, checks whose purpose is to tell you when a result should not be believed. The same idea applies to any automated system. Test each guardrail deliberately: give it an input it must catch and confirm it catches it.

How often to run it

Quarterly as a routine, and immediately after any change to tracking, consent settings, the tools a workflow depends on, or the offer. Most unhealthy systems became unhealthy on the day of a change nobody connected to them.

If you would rather not maintain the machinery yourself, AI for Marketing runs client marketing as a managed service on its own internal system, with a person accountable for the checks described here.

FAQ

Who should run the health check?

The person who owns the system's results, ideally with someone who did not build it. Builders tend to check what they expect to fail rather than what actually has.

What is the quickest single improvement for a fragile system?

Expected-output counts. For each workflow, write down how many items it should produce per run and alert when it produces fewer. It catches the most damaging failure type, the silent one.

Is a gap between ad platform and CRM conversions always a problem?

No. Attribution windows, modelled conversions and deduplication all create gaps. The problem is a gap you cannot explain, or one that changes suddenly without a known cause.

Should I switch off automation if the score is low?

Pause the parts that spend money or contact customers until the relevant checks pass. Internal drafting and research can usually carry on while you fix measurement and controls.