Keyword Monitoring: Catch Hidden Website Outages

Your uptime monitor shows all green. Your server is responding with a 200 OK status code. But your users see a blank page, a database error message, or worse, your site has been defaced. This is a "soft outage," and it's invisible to basic uptime checks.

Simple HTTP checks only validate that your server is alive and responding. They don't know if the content of that response is correct. This is where keyword monitoring, also known as content matching, becomes a critical layer in your monitoring strategy.

The "200 OK" Lie: Why Uptime Checks Aren't Enough

A standard uptime check is binary: the site is up or down. But reality is more complex. A server can be running perfectly while the application itself is broken. A 200 OK response can hide many problems:

• Application Errors: A misconfiguration or bug might cause your framework (like Laravel, Rails, or Django) to display a detailed error stack trace instead of your actual website. The status code is 200 OK, but the site is functionally down.
• Database Failures: Your web server is up, but it can't connect to the database. It renders a page with a message like "Could not connect to database".
• Empty Responses: A bug might cause your application to return a completely blank page with a 200 OK header.
• CDN Cached Errors: A Content Delivery Network (CDN) might inadvertently cache an error page and serve it globally, even after the origin server has recovered.
• Website Defacement: A security breach could replace your homepage content with something malicious. The server still responds 200 OK, but your brand reputation is at stake.

In all these cases, a basic HTTP check reports success, and you remain unaware of the problem until customers complain.

How Keyword Monitoring Works

Keyword monitoring enhances a standard HTTP check by inspecting the HTML body of the response. Instead of just looking at the status code, the monitoring service fetches the page and scans the content for a specific string.

This works in two ways:

1. Assert Text Exists (Presence Check): The monitor checks that a specific string of text is present on the page. If the text is missing, the check fails and an alert is triggered. This is useful for verifying that essential content has loaded correctly.
2. Assert Text Does Not Exist (Absence Check): The monitor checks that a specific string is not present. If the forbidden text is found, the check fails. This is ideal for catching common error messages or signs of defacement.

You can replicate this logic manually from your terminal using curl and grep.

To check if your website's footer text is present:

# Will exit with status 0 (success) if found, 1 (failure) if not
curl -s https://example.com | grep -q "© 2024 Example Inc."

To check if a common framework error is present:

# If this command finds the text, you have a problem
curl -s https://myapp.com | grep -i "fatal error"

While curl is great for spot checks, a dedicated monitoring service automates this process from multiple global locations, manages alerting, and keeps a history of results.

Practical Use Cases for Keyword Monitoring

Keyword monitoring is versatile. Here are some common use cases from basic error detection to deployment verification.

• Detecting Application Errors: The most common use case. Set up absence checks for strings like "Exception", "Error", "Warning:", "SQLSTATE", or "Notice:". This immediately catches unhandled exceptions that make it to the page.
• Verifying Critical Content: For an e-commerce site, assert that the text "Add to Cart" is present on product pages. For a blog, ensure the main <h1> title of your latest post is rendered. This confirms your application logic and database queries are working.
• Verifying Successful Deployments: Include a unique build identifier or Git commit hash in your site's HTML (often in a meta tag or comment). Set up a presence check for this new identifier after a deployment to confirm the new code is live.
• Security and Defacement Monitoring: Use a presence check for a unique, non-public string hidden in your HTML (e.g., in a comment <!-- my-secret-check-string-abc123 -->). If it's ever removed, you'll be alerted, which could be an early sign of a breach. Conversely, check for the absence of common spam or malicious keywords.

Best Practices for Choosing Keywords

Creating a reliable keyword check depends entirely on choosing the right string to look for. Follow this checklist to avoid false positives and create robust monitors.

Setting Up Keyword Monitoring with ServerInspector

While you can script this yourself, a monitoring platform makes it trivial. When setting up an HTTP check in ServerInspector, you don't need to write any code. In the check configuration, navigate to the Assertions section.

Here, you can add rules to assert that the response body Must Contain or Must Not Contain a specific string. This turns a basic uptime check into a powerful content verification tool in seconds. It will automatically check from multiple regions and handle alerting, letting you know about soft outages that a simple status code check would miss.

What About Single Page Applications (SPAs)?

Modern JavaScript frameworks like React, Vue, and Angular present a challenge. They often serve a nearly empty HTML file, and all the content is rendered dynamically by JavaScript in the user's browser. A simple curl-based check will only see the initial HTML shell, not the final content.

For these applications, you need a monitor that can execute JavaScript. This involves running a full headless browser (like Chrome) to render the page completely before checking for the keyword. This is a more advanced feature, but essential for accurately monitoring SPAs.

FAQ

Q1: Isn't keyword monitoring slow or resource-intensive? A: Not at all. For the monitoring service, scanning the downloaded text is a negligible overhead that adds only microseconds to the check after the page is fetched. The main time is spent on the network request itself.

Q2: Can I monitor for content that is not visible, like a meta tag? A: Yes. Keyword monitoring works on the raw HTML source code. Any text in the file, including meta tags, HTML comments, or inline script tags, can be monitored.

Q3: What's better: checking for text that should be there (presence) or text that shouldn't (absence)? A: It depends on your goal. For catching errors, checking for the absence of common error strings is most effective. For verifying critical functionality (like a checkout button), checking for its presence is better. A comprehensive strategy often uses both types of checks for different purposes.

Q4: How does this differ from API monitoring? A: The principle is similar, but the context is different. Keyword monitoring typically scans unstructured or semi-structured HTML meant for humans. API monitoring asserts the structure and values within highly structured data formats like JSON (e.g., checking "status": "completed") meant for machine-to-machine communication.