When was the last time you checked what your competitors changed on their website? If the answer is “during our last strategy session” or “when someone mentioned it,” you’re operating with a six-month-old map in a market that shifts weekly.
A local professional services firm learned a national franchise was entering their market—not from a press release, but from automated monitoring that caught new location pages appearing in the franchise’s sitemap. They had weeks to adjust their messaging before the national brand launched locally. When the big player arrived with their standard pitch, the local firm had already positioned against every weakness. The national brand’s marketing budget actually helped them—educating the market while the local firm captured the customers who wanted something different.
That’s the difference between reactive and proactive competitive intelligence. One approach leaves you scrambling after a competitor move becomes obvious to everyone. The other gives you time to respond strategically—or even preempt the move entirely.
The Real Cost of Not Knowing
Markets don’t wait for your next quarterly review. Competitors adjust pricing, test new messaging, launch content campaigns, and restructure their technical SEO continuously. Every week you’re not watching, you’re accumulating blind spots.
Consider what slips through when monitoring is manual and sporadic:
Pricing shifts happen quietly. A competitor raises prices twice in six weeks, responding to upstream cost increases you’re also facing. Without monitoring, you leave money on the table—or worse, you drop prices thinking you need to compete harder when the market has actually moved in your favor.
Messaging pivots signal strategic shifts. When a competitor changes their homepage headline from “The fastest solution” to “The most secure solution,” that’s not a copywriting whim. They’re repositioning, likely based on customer research or competitive pressure. That insight should inform your own positioning.
New market entries often start with small tells—a new location page in a sitemap, a job posting in a new city, a case study featuring a client in an adjacent industry. Catching these early gives you time to respond.
Content strategy changes reveal where competitors are investing. A sudden cluster of blog posts around a specific topic suggests they’re targeting new keywords or audiences. Understanding their content calendar helps you find gaps they’re ignoring.
The companies winning their categories treat competitive intelligence like they treat their financials—something tracked weekly, not annually. And you don’t need a six-figure enterprise platform to do it.
What to Actually Monitor
Effective competitor monitoring isn’t about watching everything. It’s about watching the right things with the right frequency. Here’s where to focus:
Pricing Intelligence
Track product prices, promotional banners, sale timing, and pricing page changes. For e-commerce competitors, this means monitoring individual product pages. For SaaS competitors, watch the pricing page for tier changes, feature adjustments, or new packages.
Why it matters: Pricing changes are high-signal events. A competitor raising prices suggests either confidence or cost pressure. A competitor running frequent promotions suggests inventory issues or growth pressure. Either insight informs your strategy.
Messaging and Positioning
Monitor homepage headlines, taglines, product page value propositions, and key CTAs. These are the elements competitors test and iterate on—changes here often reflect strategic decisions backed by customer research.
Why it matters: Subtle messaging shifts reveal where competitors think they can win. A shift from “powerful” to “easy” suggests they’re targeting a different buyer persona. A new emphasis on “security” over “speed” signals repositioning you should understand.
SEO and Technical Structure
Watch meta titles, meta descriptions, schema markup additions, heading structure changes, and new page templates. These technical changes often precede or accompany content strategy shifts.
Why it matters: SEO changes signal where competitors are investing for organic growth. A competitor adding FAQ schema to key pages is optimizing for featured snippets. New landing page templates suggest paid campaign expansion.
Ad Creative and Landing Pages
Track dedicated landing pages used for advertising campaigns, especially headlines, form fields, CTAs, and offer structures. These pages change frequently as competitors optimize conversion rates.
Why it matters: Landing page tests reveal what messaging resonates with the market. If a competitor’s CTA shifts from “Start Free Trial” to “Get Custom Quote,” they’re likely moving upmarket. Understanding their conversion optimization informs your own.
Content Strategy
Monitor blog feeds, sitemap changes, and new page creation. This reveals topic priorities, publishing cadence, and content investment levels.
Why it matters: Content investments take months to pay off. Knowing where competitors are investing now tells you where they expect to compete in six months.
The Open Source Stack
Enterprise competitive intelligence platforms charge thousands per month and often deliver underwhelming results—black-box algorithms, limited customization, and dashboards designed more for executive presentations than actionable intelligence.
A self-hosted stack built on open source tools costs virtually nothing beyond server time, gives you complete control over what you monitor and how, and keeps your competitive data in-house where it belongs.
Here’s what we recommend:
Changedetection.io: The Foundation
For straightforward page monitoring, Changedetection.io is remarkably capable. It tracks content changes on any webpage, highlights differences at the word or character level, and sends alerts through email, Slack, Discord, or webhooks.
Key capabilities:
- Visual CSS selector tool for monitoring specific page sections
- Built-in price tracking mode that extracts pricing metadata
- Playwright integration for JavaScript-heavy pages
- Threshold alerts (only notify when price changes exceed X%)
Deploy it with a single Docker command:
docker run -d \
--name changedetection \
-p 5000:5000 \
-v changedetection-data:/datastore \
ghcr.io/dgtlmoon/changedetection.io
For pages that require JavaScript rendering, add the Playwright container:
# docker-compose.yml
version: '3'
services:
changedetection:
image: ghcr.io/dgtlmoon/changedetection.io
ports:
- "5000:5000"
volumes:
- changedetection-data:/datastore
environment:
- PLAYWRIGHT_DRIVER_URL=ws://playwright-chrome:3000
depends_on:
- playwright-chrome
playwright-chrome:
image: browserless/chrome
restart: unless-stopped
environment:
- SCREEN_WIDTH=1920
- SCREEN_HEIGHT=1080
volumes:
changedetection-data:
This handles 80% of competitor monitoring needs with minimal configuration.
Scrapy: Structured Data at Scale
When you need to extract structured data across many pages—monitoring an entire product catalog, tracking prices across dozens of SKUs, or auditing technical SEO elements site-wide—Scrapy is the right tool.
Here’s a practical spider for monitoring competitor pricing:
# competitor_prices.py
import scrapy
import json
from datetime import datetime
from pathlib import Path
class CompetitorPriceSpider(scrapy.Spider):
name = "competitor_prices"
# Define competitor product URLs to monitor
start_urls = [
"https://competitor.com/product/widget-pro",
"https://competitor.com/product/widget-enterprise",
"https://competitor.com/product/widget-starter",
]
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.prices_file = Path("price_history.json")
self.previous_prices = self._load_previous_prices()
self.current_prices = {}
self.changes = []
def _load_previous_prices(self):
if self.prices_file.exists():
return json.loads(self.prices_file.read_text())
return {}
def parse(self, response):
# Adjust selectors for your target site
product_name = response.css("h1.product-title::text").get()
price_text = response.css("span.price::text").get()
# Clean and parse price
if price_text:
price = float(price_text.replace("$", "").replace(",", "").strip())
else:
price = None
url = response.url
self.current_prices[url] = {
"product": product_name,
"price": price,
"timestamp": datetime.now().isoformat()
}
# Check for changes
if url in self.previous_prices:
old_price = self.previous_prices[url].get("price")
if old_price and price and old_price != price:
change_pct = ((price - old_price) / old_price) * 100
self.changes.append({
"product": product_name,
"url": url,
"old_price": old_price,
"new_price": price,
"change_percent": round(change_pct, 2)
})
yield {
"url": url,
"product": product_name,
"price": price,
}
def closed(self, reason):
# Save current prices for next comparison
self.prices_file.write_text(json.dumps(self.current_prices, indent=2))
# Output changes
if self.changes:
print("\n=== PRICE CHANGES DETECTED ===")
for change in self.changes:
direction = "↑" if change["change_percent"] > 0 else "↓"
print(f"{change['product']}: ${change['old_price']} → ${change['new_price']} ({direction}{abs(change['change_percent'])}%)")
Run it on a schedule with cron:
# Run daily at 6 AM
0 6 * * * cd /path/to/project && scrapy crawl competitor_prices -o prices_$(date +\%Y\%m\%d).json
Handling Anti-Bot Protection
Modern websites increasingly deploy bot detection. For reliable monitoring, you’ll often need residential proxies. We use BrightData, but any reputable proxy provider works:
# settings.py for Scrapy
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}
# Rotate through proxy pool
ROTATING_PROXY_LIST = [
'http://user:pass@proxy1.brightdata.com:22225',
'http://user:pass@proxy2.brightdata.com:22225',
]
# Respect rate limits
DOWNLOAD_DELAY = 2
RANDOMIZE_DOWNLOAD_DELAY = True
CONCURRENT_REQUESTS_PER_DOMAIN = 2
# Rotate user agents
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
AI-Powered Change Interpretation
Raw diffs are useful, but context is better. When a change is detected, use an LLM to interpret what it means:
# analyze_change.py
import anthropic
from pathlib import Path
def analyze_competitor_change(old_content: str, new_content: str, page_type: str) -> str:
"""Use Claude to interpret a competitor website change."""
client = anthropic.Anthropic()
prompt = f"""Analyze this competitor website change and provide strategic insights.
Page type: {page_type}
PREVIOUS CONTENT:
{old_content}
NEW CONTENT:
{new_content}
Provide:
1. A one-sentence summary of what changed
2. What this change likely signals about their strategy
3. Whether this warrants immediate attention or is routine
4. Any recommended actions for our team
Be concise and focus on actionable intelligence."""
message = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=500,
messages=[
{"role": "user", "content": prompt}
]
)
return message.content[0].text
def analyze_pricing_change(product: str, old_price: float, new_price: float) -> str:
"""Specific analysis for pricing changes."""
client = anthropic.Anthropic()
change_pct = ((new_price - old_price) / old_price) * 100
direction = "increase" if change_pct > 0 else "decrease"
prompt = f"""A competitor changed their pricing:
Product: {product}
Previous price: ${old_price}
New price: ${new_price}
Change: {direction} of {abs(change_pct):.1f}%
What does this pricing move likely signal? Consider:
- Market conditions that might drive this change
- Potential strategic motivations
- How we should respond (if at all)
Be concise and actionable."""
message = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=300,
messages=[
{"role": "user", "content": prompt}
]
)
return message.content[0].text
if __name__ == "__main__":
# Example usage
analysis = analyze_pricing_change(
product="Enterprise Widget",
old_price=299.00,
new_price=349.00
)
print(analysis)
Validating Selectors with AI
Websites change structure frequently, breaking scrapers. Use AI to validate and suggest selectors:
# selector_validator.py
import anthropic
import requests
from bs4 import BeautifulSoup
def validate_selector(url: str, target_data: str, current_selector: str) -> dict:
"""Check if a CSS selector still works and suggest fixes if not."""
# Fetch the page
response = requests.get(url, headers={
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
})
soup = BeautifulSoup(response.content, 'html.parser')
# Test current selector
result = soup.select_one(current_selector)
if result and result.get_text(strip=True):
return {
"status": "working",
"selector": current_selector,
"extracted": result.get_text(strip=True)[:100]
}
# Selector broken - ask AI for help
# Get a sample of the HTML structure
html_sample = str(soup)[:8000] # Truncate for context limits
client = anthropic.Anthropic()
prompt = f"""A CSS selector stopped working on a competitor website.
Target data: {target_data}
Broken selector: {current_selector}
Here's the current HTML structure (truncated):
{html_sample}
Suggest 2-3 CSS selectors that would likely extract the {target_data}.
Return ONLY the selectors, one per line, most likely first."""
message = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=200,
messages=[
{"role": "user", "content": prompt}
]
)
suggested = message.content[0].text.strip().split('\n')
# Test suggested selectors
for selector in suggested:
selector = selector.strip()
if selector:
test_result = soup.select_one(selector)
if test_result and test_result.get_text(strip=True):
return {
"status": "fixed",
"old_selector": current_selector,
"new_selector": selector,
"extracted": test_result.get_text(strip=True)[:100]
}
return {
"status": "needs_manual_review",
"selector": current_selector,
"suggestions": suggested
}
Orchestrating with n8n
For teams that prefer visual workflow builders, n8n provides a self-hosted alternative to Zapier that integrates well with custom scripts:
{
"name": "Competitor Price Alert",
"nodes": [
{
"name": "Schedule",
"type": "n8n-nodes-base.cron",
"parameters": {
"cronExpression": "0 6 * * *"
}
},
{
"name": "Run Scrapy",
"type": "n8n-nodes-base.executeCommand",
"parameters": {
"command": "cd /opt/monitoring && scrapy crawl competitor_prices -o /tmp/prices.json"
}
},
{
"name": "Read Results",
"type": "n8n-nodes-base.readBinaryFile",
"parameters": {
"filePath": "/tmp/prices.json"
}
},
{
"name": "Check for Changes",
"type": "n8n-nodes-base.function",
"parameters": {
"functionCode": "// Compare to previous run and flag changes\nconst data = JSON.parse($input.first().binary.data.toString());\nconst changes = data.filter(item => item.price_changed);\nreturn changes.map(c => ({ json: c }));"
}
},
{
"name": "Alert Slack",
"type": "n8n-nodes-base.slack",
"parameters": {
"channel": "#competitor-watch",
"text": "Price change detected: {{ $json.product }} - ${{ $json.old_price }} → ${{ $json.new_price }}"
}
}
]
}
For the Technically Inclined
Monitoring SEO Changes
Track meta tags, schema markup, and heading structure across competitor pages:
# seo_monitor.py
import scrapy
import json
import hashlib
from datetime import datetime
class SEOMonitorSpider(scrapy.Spider):
name = "seo_monitor"
start_urls = [
"https://competitor.com/",
"https://competitor.com/product",
"https://competitor.com/pricing",
"https://competitor.com/about",
]
def parse(self, response):
# Extract SEO elements
seo_data = {
"url": response.url,
"timestamp": datetime.now().isoformat(),
"title": response.css("title::text").get(),
"meta_description": response.css('meta[name="description"]::attr(content)').get(),
"canonical": response.css('link[rel="canonical"]::attr(href)').get(),
"h1": response.css("h1::text").getall(),
"h2": response.css("h2::text").getall(),
"schema_types": self._extract_schema_types(response),
"robots": response.css('meta[name="robots"]::attr(content)').get(),
}
# Generate hash for change detection
content_hash = hashlib.md5(
json.dumps(seo_data, sort_keys=True).encode()
).hexdigest()
seo_data["content_hash"] = content_hash
yield seo_data
def _extract_schema_types(self, response):
"""Extract JSON-LD schema types from page."""
schemas = []
for script in response.css('script[type="application/ld+json"]::text').getall():
try:
data = json.loads(script)
if isinstance(data, dict):
schemas.append(data.get("@type", "Unknown"))
elif isinstance(data, list):
schemas.extend(item.get("@type", "Unknown") for item in data if isinstance(item, dict))
except json.JSONDecodeError:
continue
return schemas
Sitemap Monitoring for New Content
Detect when competitors add new pages:
# sitemap_monitor.py
import requests
import xml.etree.ElementTree as ET
from datetime import datetime
from pathlib import Path
import json
def fetch_sitemap_urls(sitemap_url: str) -> set:
"""Extract all URLs from a sitemap."""
response = requests.get(sitemap_url)
root = ET.fromstring(response.content)
# Handle namespace
ns = {'sm': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
urls = set()
for url in root.findall('.//sm:url/sm:loc', ns):
urls.add(url.text)
# Handle sitemap index files
for sitemap in root.findall('.//sm:sitemap/sm:loc', ns):
urls.update(fetch_sitemap_urls(sitemap.text))
return urls
def check_for_new_pages(competitor_name: str, sitemap_url: str) -> dict:
"""Compare current sitemap to previous run, report new URLs."""
history_file = Path(f"sitemap_history_{competitor_name}.json")
current_urls = fetch_sitemap_urls(sitemap_url)
if history_file.exists():
history = json.loads(history_file.read_text())
previous_urls = set(history.get("urls", []))
else:
previous_urls = set()
new_urls = current_urls - previous_urls
removed_urls = previous_urls - current_urls
# Save current state
history_file.write_text(json.dumps({
"urls": list(current_urls),
"last_checked": datetime.now().isoformat()
}, indent=2))
return {
"competitor": competitor_name,
"total_pages": len(current_urls),
"new_pages": list(new_urls),
"removed_pages": list(removed_urls),
"checked_at": datetime.now().isoformat()
}
if __name__ == "__main__":
result = check_for_new_pages(
competitor_name="acme-corp",
sitemap_url="https://competitor.com/sitemap.xml"
)
if result["new_pages"]:
print(f"🆕 {len(result['new_pages'])} new pages detected:")
for url in result["new_pages"][:10]: # Show first 10
print(f" - {url}")
if result["removed_pages"]:
print(f"🗑️ {len(result['removed_pages'])} pages removed:")
for url in result["removed_pages"][:10]:
print(f" - {url}")
Complete Notification Pipeline
Tie everything together with a unified alerting system:
# notify.py
import requests
import json
from dataclasses import dataclass
from typing import Optional
@dataclass
class Alert:
competitor: str
alert_type: str # price_change, seo_change, new_content, messaging_change
severity: str # high, medium, low
summary: str
details: dict
url: Optional[str] = None
def send_slack_alert(alert: Alert, webhook_url: str):
"""Send formatted alert to Slack."""
severity_emoji = {
"high": "🚨",
"medium": "⚠️",
"low": "📋"
}
type_emoji = {
"price_change": "💰",
"seo_change": "🔍",
"new_content": "📝",
"messaging_change": "💬"
}
blocks = [
{
"type": "header",
"text": {
"type": "plain_text",
"text": f"{severity_emoji[alert.severity]} {type_emoji[alert.alert_type]} Competitor Alert"
}
},
{
"type": "section",
"fields": [
{"type": "mrkdwn", "text": f"*Competitor:*\n{alert.competitor}"},
{"type": "mrkdwn", "text": f"*Type:*\n{alert.alert_type.replace('_', ' ').title()}"}
]
},
{
"type": "section",
"text": {"type": "mrkdwn", "text": f"*Summary:*\n{alert.summary}"}
}
]
if alert.url:
blocks.append({
"type": "section",
"text": {"type": "mrkdwn", "text": f"<{alert.url}|View Page>"}
})
if alert.details:
details_text = "\n".join(f"• {k}: {v}" for k, v in alert.details.items())
blocks.append({
"type": "section",
"text": {"type": "mrkdwn", "text": f"*Details:*\n{details_text}"}
})
payload = {"blocks": blocks}
requests.post(webhook_url, json=payload)
def send_email_alert(alert: Alert, smtp_config: dict):
"""Send alert via email."""
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
msg = MIMEMultipart('alternative')
msg['Subject'] = f"[{alert.severity.upper()}] {alert.competitor}: {alert.alert_type.replace('_', ' ').title()}"
msg['From'] = smtp_config['from']
msg['To'] = smtp_config['to']
html = f"""
<html>
<body>
<h2>Competitor Alert: {alert.competitor}</h2>
<p><strong>Type:</strong> {alert.alert_type.replace('_', ' ').title()}</p>
<p><strong>Severity:</strong> {alert.severity}</p>
<p><strong>Summary:</strong> {alert.summary}</p>
{'<p><a href="' + alert.url + '">View Page</a></p>' if alert.url else ''}
<h3>Details</h3>
<ul>
{''.join(f'<li><strong>{k}:</strong> {v}</li>' for k, v in alert.details.items())}
</ul>
</body>
</html>
"""
msg.attach(MIMEText(html, 'html'))
with smtplib.SMTP(smtp_config['host'], smtp_config['port']) as server:
server.starttls()
server.login(smtp_config['user'], smtp_config['password'])
server.send_message(msg)
A Note on Ethics and Legality
Monitoring publicly available information on competitor websites is standard business practice. That said, respect boundaries:
- Honor robots.txt directives for crawl frequency and restricted areas
- Don’t overwhelm competitor servers with aggressive scraping—use reasonable delays
- Never attempt to access authenticated or restricted content
- Focus on public pages, public pricing, and public content
If a competitor explicitly blocks your IP or employs anti-bot measures, consider whether the information is truly “public” and whether pursuing it aligns with how you’d want competitors to treat your own site.
When to DIY vs. Hire an Agency
Self-hosted monitoring works well when you have:
- Technical resources to maintain scripts and infrastructure
- A manageable number of competitors (3-5)
- Relatively stable competitor website structures
- Time to review alerts and derive insights
Consider agency support when you need:
- Monitoring at scale (many competitors, many pages, many data points)
- Structured reporting and strategic analysis layered on top of raw monitoring
- Integration with broader competitive intelligence and market research
- Someone else to handle the maintenance and keep everything running
A reasonable guideline: allocate roughly 33% of your market research budget to ongoing competitive intelligence. That might mean funding internal tools and time, or it might mean partnering with specialists who handle the technical infrastructure while you focus on strategic response.
Start Watching
Your competitors made changes to their website this week. Maybe they adjusted pricing. Maybe they tested new messaging. Maybe they published content targeting keywords you thought were yours.
You can find out next quarter when someone mentions it in a meeting. Or you can know by Monday morning.
The tools are free. The infrastructure cost is negligible. The only investment is the hour it takes to set up your first monitor.
Want more technical marketing insights? Follow our socials and read our blog posts.