C:\Users\Administrator\Desktop>python scrape_myprotein.py
Target: Playwright Headless Chromium
Sheet1
Data Summary
PivotTable
| # | A: Title | B: Price | C: Reviews | D: Product Link |
|---|
DOM Selector Cascade & Anti-Bot Resilience
Modern e-commerce sites dynamically randomize or hash class names across build releases. Hardcoded single CSS selectors trigger silent scraper failures. The Playwright engine uses ordered selector cascades and stealth browser context settings to achieve 99.9% extraction uptime.
Ordered CSS / XPath Selector Cascades
# 1. Product Title Cascade
title_selectors = [
".athenaProductBlock_title", # Standard class
"h3", # Semantic header
"[data-component='product-title']", # Semantic attribute
"a[data-product-title]" # Data attribute
]
# 2. Product Price Cascade
price_selectors = [
".athenaProductBlock_priceValue", # Direct price node
"[data-component='product-price']", # Component container
".athenaProductBlock_price", # Parent price row
"span[class*='price']" # Substring match
]
# 3. Product Reviews Cascade
review_selectors = [
".athenaProductBlock_reviewCount", # Count badge: '(35)'
"[data-component='product-reviews']",# Review container
".athenaProductBlock_rating" # Star rating row
]
Headless Chromium Stealth Config
# Stealth launch arguments (bypasses navigator.webdriver flags)
browser = playwright.chromium.launch(
headless=True,
args=[
"--disable-blink-features=AutomationControlled",
"--no-sandbox",
"--disable-dev-shm-usage",
]
)
# Realistic Desktop Context Configuration
context = browser.new_context(
viewport={"width": 1920, "height": 1080},
user_agent=(
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36"
)
)
# Strip automation telemetry from window
context.add_init_script(
"Object.defineProperty(navigator, 'webdriver', {get: () => undefined})"
)
Deconstructing the Clue: "Use AI to write it."
Why did the challenge specify "Here's your clue: Use AI to write it"? Writing web scrapers by hand leads to fragile single-selector code, unhandled cookie overlays, and encoding bugs. AI prompt engineering allows engineers to synthesize self-healing, multi-tier scrapers with robust edge-case handling.
❌ The Naive User Prompt (Fails in Production)
"Write me a python script to scrape products from myprotein into a csv file."
- Uses
requests.get()which gets blocked by Cloudflare or returns empty CSR root shell. - Fails to dismiss the OneTrust cookie modal, occluding elements.
- Hardcodes a single volatile class name like
.title-xyzthat breaks in 2 weeks. - Exports CSV with plain ASCII/UTF-8, turning British Pounds (
£) and registered trademarks (®) into mojibake (€") in Microsoft Excel.
✅ The Master Production Prompt (Robust)
Write a production-grade Python script using Playwright to scrape an e-commerce catalog (e.g. MyProtein):
1. Initialize Chromium with headless=True, realistic desktop User-Agent, and disable AutomationControlled flag.
2. Navigate to target catalog and handle cookie consent overlays (#onetrust-accept-btn-handler).
3. Query product cards using selector cascades (Title, Price, Review Count, and canonical Product URL).
4. Resolve relative URLs to absolute HTTP targets via urllib.parse.urljoin.
5. Capture a diagnostic screenshot (screenshot.png).
6. Export structured records to CSV using utf-8-sig encoding so Excel renders currency (£) and glyphs (®) without corruption.
7. Include offline HTML fixture fallback for isolated CI/CD testing.
Playwright Sync vs Async Comparison
# Playwright offers both Synchronous and Asynchronous execution models:
from playwright.sync_api import sync_playwright # Best for CLI scripts, cron jobs, simple loops
from playwright.async_api import async_playwright # Best for FastAPI endpoints, high-concurrency 100+ tab scrapers
Local Workstation & Cloud VM Deployment
Playwright drives headless browsers directly on macOS, Windows, Linux VMs (Ubuntu/Debian), and Docker containers. When deploying to a cloud virtual machine, system dependencies (e.g. libnss3, libasound2) must be installed alongside the Python package.
Ubuntu 22.04 / 24.04 Cloud VM Setup
# 1. Update OS packages and Python
sudo apt update && sudo apt install -y python3-pip python3-venv
# 2. Clone repository and create virtual environment
python3 -m venv venv
source venv/bin/activate
# 3. Install Playwright & OS system libraries
pip install playwright
sudo playwright install-deps chromium
playwright install chromium
# 4. Run scraper in headless mode
python scrape_myprotein.py --headless --output products.csv
Production Docker Container Execution
# Build and run using the official Microsoft Playwright image
docker build -t ecommerce-scraper:latest .
# Run and mount current directory to extract CSV
docker run --rm -v $(pwd):/app ecommerce-scraper:latest
# Schedule daily execution via Linux crontab
0 3 * * * cd /opt/scraper && ./venv/bin/python scrape_myprotein.py
Automated 8-Point In-Browser Diagnostic Test Suite
Automated validation suite verifying DOM card structure, cookie modal bypass, 4-field extraction completeness, URL canonicalization, price parsing, and RFC 4180 CSV serialization with UTF-8 BOM encoding.
Tests: 8 Total
0 Passed
0 Failed
8 Pending
Pending
1. Product Card Locator & Grid Resolution
Verifies product card elements are present in the DOM hierarchy.
Pending
2. Cookie Consent Overlay Dismissal
Validates simulated Playwright click on cookie consent modal.
Pending
3. Product Title Extraction & Unicode Glyph Safety
Validates title string extraction and trademark symbol preservation (®).
Pending
4. Price Extraction & Currency Symbol Normalization
Checks price presence, numeric decimals, and British Pound (£) symbol.
Pending
5. Review Count Numeric Parsing
Confirms review strings '(35)', '(29454)' are cleaned to plain integers.
Pending
6. Canonical Absolute URL Resolution
Ensures relative path '/p/...' is normalized to fully-qualified HTTPS link.
Pending
7. CSV Header & RFC 4180 Row Alignment
Validates CSV column order [Title, Price, Reviews, Product Link].
Pending
8. Excel UTF-8 BOM Signature Verification
Verifies \xef\xbb\xbf byte header to prevent Windows Excel mojibake.