Commit Graph

5 Commits

Author SHA1 Message Date
wasrusgen
1a57374020 parsers: better image extraction — real product photos in report cards
CITILINK:
- Now reads data-src / data-original / srcset / src in priority order
- srcset → picks largest size variant (last in comma-list)
- Filters only _next/static/images (placeholder) and 'placeholder' in URL
- Accepts cs.citilink.ru / c.citilink.ru / images.citilink.ru product photos

ЯНДЕКС.МАРКЕТ:
- Collects all img attrs (data-src, data-original, srcset, data-srcset, src)
- Prefers avatars.mds.yandex.net (real product CDN), skips yastatic (icons/logos)
- Auto-appends /300x300 suffix to avatars.mds URLs without size

ENRICH_ONE (aggregator):
- Image picked by source priority: yamarket > wb > ozon > citilink > dns
- Yamarket photos are cleanest (avatars.mds.yandex.net)
- WB has product photos via basket-XX.wbbasket.ru
2026-05-11 23:43:25 +03:00
wasrusgen
c97b8dce3c parsers: skip sponsored/ad URLs (cpc/sponsored=1) — they expire in 2-3 hours
User reported clicking matrix prices led to 'Произошла ошибка!' on OZON home page.
Cause: parsers captured /product/?sponsored=1&cpc=Jtiito95... links that died after few hours.

Fix:
- ozon.py: skip href with 'sponsored=1', '/promo/', 'cpc='. Strip query string from final URL.
- yamarket.py: skip 'sponsored=1', 'cpc=', 'advUuid' (Я.Маркет sponsored marker)
- citilink.py: strip query string from final URL (defensive)

Now matrix links go to canonical product pages that don't expire.
2026-05-11 17:20:59 +03:00
wasrusgen
44281b1e07 citilink: dedup by product ID + filter Next.js placeholder images 2026-05-11 13:59:07 +03:00
wasrusgen
c5f662f53d citilink: rewrite parser to walk up from a[href*=/product/] (CSS-in-JS resistant) 2026-05-11 13:57:18 +03:00
wasrusgen
e8b487891f backend: working parsers — OZON + Citilink (DOM via Playwright) + WB
DIAGNOSTIC RESULTS:
- OZON: 19 product links via Playwright on naked VPS-IP ✓
- Citilink: 112 data-meta-name Snippets ✓
- Wildberries: JSON API works with delays ✓
- Я.Маркет, DNS: blocked by ASN (need residential proxy)

OZON PARSER:
- Pure Playwright DOM (composer-api dropped — was blocked)
- Selects a[href*='/product/'], walks up to card div, extracts title/price/img
- Filters fake 'titles' like Распродажа, Скидка

CITILINK PARSER (new):
- Selects [data-meta-name*='Snippet'] or ProductCard markers
- Multiple title selectors fallback chain
- Filters out non-product hits

PARSERS/__init__.py:
- DEFAULT_SOURCES = (ozon, citilink, wb) — all work without proxy
- Я.Маркет, DNS kept but not default — usable when residential proxy added

NEW ENDPOINT:
- GET /api/parse_citilink?q=...&limit=N
2026-05-11 13:53:07 +03:00