5 Commits

Author	SHA1	Message	Date
wasrusgen	d84a53f91d	wb: skip proxy pool (use direct VPS-IP — residential were rate-limited)	2026-05-11 23:01:34 +03:00
wasrusgen	e9b0db6772	wb: API v9 → v18 (WB сменил endpoint и структуру) + brand+category fallback query ... DISCOVERED in real test: - WB API v9 (/exactmatch/ru/common/v9/search) теперь возвращает только метаданные (name, query, shardKey, filters, search_result={}) — products пусто - WB API v18 (/exactmatch/ru/common/v18/search) — рабочий Структура: {metadata, products, total} — products НА ВЕРХНЕМ уровне (не data.products) - Подтверждено: query='Haier холодильник' → 100 products via v18 CHANGES: 1. _SEARCH_URL → v18 endpoint 2. Парсинг products: сначала data.products (legacy fallback), потом products top-level 3. _build_item: цены теперь читаются из sizes[].price.{product, total, basic} (v18 формат), с fallback на priceU/salePriceU (v9 legacy) 4. _generate_query_variants: добавлен brand+category fallback ('Bosch холодильник' если не нашли по модели) TEST: Haier холодильник → 100 results (first: 'Холодильник двухкамерный C2F619CFU1')	2026-05-11 22:59:14 +03:00
wasrusgen	ca342c0641	ai+report: deeper analysis — required pros/cons, category insights, source visibility ... AI PROMPT (ai.py): - Requires minimum 3 pros + 2 cons per model with NUMBERS (36 dB, 463 L, A++, не 'тихий/большой') - New field 'reasoning' — 1-sentence why-this-model justification - New per-category 'analysis' — 2-3 sentences about trade-offs - Strict rules: no fake article numbers, account for parallel-import price markup - Russian market 2026 awareness: Haier/Korting up, Bosch/Siemens ⚠ TELEGRAM FORMAT (main.py): - Renders category analysis as italic prelude - Lists pros/cons as bullet lists (up to 4 pros, 3 cons) - Shows '🛒 Нашли в: OZON · Citilink · WB' line listing successful sources - Rating + reviews + stores count line: '📊 ★ 4.7 · 1242 отзыв. · 12 магаз.' - Direct link to best store: '🔗 Открыть в магазине' WB PARSER: - Generates 3 query variants per request: full → brand+model → model only - Increases hit rate when AI search_query is too verbose - First non-empty variant wins MINIAPP REPORT (podbor.js + podbor.css): - Category analysis block above models (italic, walnut left-border) - Pros block: green tinted bg, bullet list, header 'Плюсы' - Cons block: terracotta tinted bg, bullet list, header 'Минусы' - Reasoning chip: 💡 italic in warm background - Source badges with per-store price '<store> · 89 990 ₽' - Color-coded source links: OZON blue, Citilink yellow, WB pink, Я.Маркет red, DNS orange - 'X магазинов нашли товар' header + plural fix - '— не найден' fallback if 0 sources PREVIEW (preview-report.html): - Mock updated with Haier as flagship (more relevant for 2026 RF) - Shows analysis, reasoning, source spread (4 stores with different prices)	2026-05-11 14:34:08 +03:00
wasrusgen	d5f290bd0a	backend: Playwright + Chromium for JS-rendered sites (Я.Маркет, OZON fallback) ... DOCKERFILE: - + Chromium system deps (libnss3, libxkbcommon0, libgbm1, libgtk-3-0, etc.) - + RUN python -m playwright install chromium (~150MB) - + ENV PLAYWRIGHT_BROWSERS_PATH REQUIREMENTS: - + playwright >= 1.45 PARSERS: - new playwright_engine.py — singleton browser, isolated context per request, blocks images/fonts/CSS to save memory, waits for selector + JS hydration - yamarket.py — rewritten to use Playwright (Я.Маркет is React SPA) - ozon.py — Playwright fallback when composer-api returns challenge (403) - wb.py — exponential backoff on 429, still uses direct HTTP (JSON API, no JS needed) STRATEGY (Hybrid Path C): - Я.Маркет: Playwright (rendering JS) - OZON: composer-api first, Playwright fallback - WB: direct HTTP with backoff (JSON API, fast) - DNS: kept but lower priority (Qrator hard to crack) - No more proxy needed for primary path DEPLOY: removed PROXY_STATIC_LIST from .env, expect ~5min for first build (Chromium download)	2026-05-11 13:25:05 +03:00
wasrusgen	82425dbd88	backend: Proxy6 pool + parsers WB / OZON / Я.Маркет / DNS ... PROXY POOL (app/proxy_pool.py): - Loads active proxies from Proxy6.net API every 10 min - Random rotation per request via proxied_client(timeout, headers) - Graceful fallback to direct HTTP if PROXY6_TOKEN not set - Config: PROXY6_TOKEN env var PARSERS (app/parsers/): - dns.py — refactored to use proxy_pool with retry+rotation on Qrator block - wb.py — Wildberries JSON API (search.wb.ru), retries on 429 - ozon.py — OZON composer-api JSON (widgetStates extraction) - yamarket.py — Я.Маркет HTML + embedded JSON parser - __init__.py — enrich_one() fans out to all sources, aggregates min/max prices, max rating, sum reviews - enrich_models() — batch enrich for AI by_category output NEW DIAGNOSTIC ENDPOINTS (main.py): - GET /api/parse_wb?q=...&limit=N - GET /api/parse_ozon?q=...&limit=N - GET /api/parse_yamarket?q=...&limit=N - GET /api/parse_all?q=... — fan-out + aggregate - GET /api/proxy_status — pool diagnostics (count, token configured, age) PODBOR (main.py): - _enrich_ai_with_dns -> _enrich_ai_marketplaces (uses all sources) DEPLOY: needs PROXY6_TOKEN in /opt/zov-tech/deploy/.env on VPS, then docker compose build + up -d backend	2026-05-11 12:18:04 +03:00