Monitoring web page changes with Cloud Functions

Introduction

I wanted to know when a few web pages changed: a price list, a page of terms and conditions, a list of job openings. Browser extensions exist for this, but they only run while the browser is open, and they don't keep a history.

So I built a small watcher on Google Cloud, which costs nothing at this scale (everything stays within the free tier):

Cloud Scheduler publishes a message every hour on a Pub/Sub topic, with the URL to watch and a CSS selector
The message triggers a Cloud Function, which downloads the page and keeps only the part matching the selector
The function hashes that content and checks in Firestore whether the same hash was already seen
If the content is new, it saves a full snapshot to Cloud Storage and records the hash in Firestore

The selector matters: a full page changes all the time (ads, timestamps, session tokens), so comparing whole pages would raise an alert at every run. Watching only div.content or table#prices gives far fewer false positives.

Data model

Each watched URL gets one Firestore document, whose ID is the MD5 hash of the URL. Under it, a snapshots sub-collection holds one document per distinct version of the content, whose ID is the MD5 hash of that content.

pages/                              (collection)
  {md5(url)}                        (document: url, updated_at)
    snapshots/                      (sub-collection)
      {md5(content)}                (document: time)

Checking whether a version was already seen is a single document read, with no query: the ID of the document is the content itself, hashed. And since the check looks at every past version, not just the last one, a page that switches back to an earlier version doesn't create a duplicate snapshot.

The snapshots themselves go to a Cloud Storage bucket, named {md5(url)}_{timestamp}, so that Firestore only stores small documents.

The function

import base64
import json
import hashlib
from datetime import datetime

import requests
from bs4 import BeautifulSoup
from google.cloud import firestore
from google.cloud import storage


def md5(string):
    return hashlib.md5(string.encode()).hexdigest()


def now():
    return datetime.now().strftime("%Y%m%d%H%M%S")


class Monitor:
    def __init__(self, collection, bucket, url, selector):
        self.url = url
        self.selector = selector
        self.collection = collection
        self.bucket = bucket
        self.db = firestore.Client()
        self.gcs = storage.Client()
        self.page_ref = self.db.collection(collection).document(md5(url))

    def fetch_content(self):
        """Download the page and keep only the part matching the selector."""
        r = requests.get(self.url, timeout=30)
        if r.status_code != requests.codes.ok:
            return None
        result = BeautifulSoup(r.text, "lxml").select(self.selector)
        return str(result[0]) if result else None

    def already_seen(self, content):
        """True if this exact content was already saved for this URL."""
        return self.page_ref.collection("snapshots").document(md5(content)).get().exists

    def save(self, content):
        """Store the snapshot in Cloud Storage and its hash in Firestore."""
        blob = self.gcs.bucket(self.bucket).blob(f"{md5(self.url)}_{now()}")
        blob.upload_from_string(content)
        self.page_ref.set({"url": self.url, "updated_at": now()})
        self.page_ref.collection("snapshots").document(md5(content)).set({"time": now()})

    def run(self):
        content = self.fetch_content()
        if content is None:
            print(f"Could not fetch '{self.url}'")
        elif self.already_seen(content):
            print(f"No change on '{self.url}'")
        else:
            self.save(content)
            print(f"Change detected on '{self.url}'")


def main(event, context):
    """Entry point, triggered by a Pub/Sub message."""
    if "data" not in event:
        return
    params = json.loads(base64.b64decode(event["data"]).decode("utf-8"))
    Monitor(
        params["collection"],
        params["bucket"],
        params["url"],
        params["selector"],
    ).run()

The function receives the Pub/Sub message as event, with its payload base64-encoded in event["data"]. Putting the URL and the selector in the message, rather than in the code, means one function can watch any number of pages: each page is simply one more scheduled job.

Alongside main.py, the requirements.txt:

requests
beautifulsoup4
lxml
google-cloud-firestore
google-cloud-storage

Deployment

Create the topic, deploy the function with that topic as its trigger, then schedule one job per page to watch:

# Pub/Sub topic
gcloud pubsub topics create webmonitor

# Cloud Function, triggered by the topic
gcloud functions deploy webmonitor \
    --runtime python37 \
    --entry-point main \
    --trigger-topic webmonitor \
    --region europe-west1

# One scheduled job per page, every hour
gcloud scheduler jobs create pubsub watch-prices \
    --schedule "0 * * * *" \
    --topic webmonitor \
    --message-body '{"collection": "pages", "bucket": "my-webmonitor-snapshots", "url": "https://example.com/prices", "selector": "table#prices"}'

The schedule uses the cron syntax, so 0 9 * * 1 would check every Monday at 9am instead.

Limits

Pages built with JavaScript come back nearly empty with requests: the content has to be rendered first, with a headless browser, which no longer fits in a small function.
The watcher records changes but doesn't warn anyone. An email or a chat message can be added in save(), or with a second function triggered by new files in the bucket.
Comparing hashes says that the content changed, not what changed. To see the difference, download the two snapshots from the bucket and compare them with diff.

##Introduction

##Data model

##The function

##Deployment

##Limits

Introduction

Data model

The function

Deployment

Limits