Project Background
scraping_coupang is a scraping project that automatically collects product data from Coupang. It was developed over roughly two weeks in December 2024 – January 2025, across 9 commits.
Scraping Pipeline
flowchart TD
A[카테고리 URL] --> B[상품 목록 페이지]
B --> C{페이지네이션}
C -->|다음 페이지| B
C -->|마지막| D[상품 상세 페이지]
D --> E[데이터 추출]
E --> F[정제 및 저장]
F --> G[CSV/DB]
Collected Data Fields
| Field | Description |
|---|---|
| Product name | Product title |
| Price | Current price, discounted price |
| Review count | Number of reviews |
| Rating | Star rating |
| Category | Category path |
| Image | Product image URL |
| Seller | Seller information |
Anti-Scraping Countermeasures
Coupang actively blocks scraping. The mitigations used:
- User-Agent randomization
- Random delay between requests
- Selenium/Playwright integration
- IP rotation (when needed)
import random
import time
USER_AGENTS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...",
]
def get_page(url):
headers = {"User-Agent": random.choice(USER_AGENTS)}
time.sleep(random.uniform(1.0, 3.0))
return requests.get(url, headers=headers)
Usage
The collected data was used as training data for category classification in the prj_category_llm project. This was the first stage of a standard ML pipeline: data collection → preprocessing → training.
Development was concentrated between 12/31 and 1/14, with 9 commits over approximately two weeks.