Spaces:

Marc034
/

al-commerce-agent

Sleeping

App Files Files Community

al-commerce-agent / README.md

Marc034

Update README.md

feb0a08 verified 5 months ago

preview code

Raw

History Blame Contribute Delete

16.5 kB

metadata

title: Al Commerce Agent
emoji: 🛍️
colorFrom: blue
colorTo: purple
sdk: docker
app_port: 7860

Al — AI-Powered Commerce Agent

An AI-powered multimodal shopping assistant for an electronics commerce website, inspired by Amazon Rufus. A single unified agent handles general conversation, text-based product recommendations, and image-based product search — all within one endpoint.

Author: Marco Wang

Features

Feature	Example	How It Works
General Conversation	"What's your name?", "What can you do?"	Intent detection bypasses retrieval; LLM responds conversationally
Text-Based Recommendation	"Recommend me wireless earbuds under $100"	MiniLM encoder → FAISS text index → Cross-Encoder re-rank → LLM reasoning
Image-Based Search	Upload a photo of headphones	OpenCLIP encoder → FAISS visual index → Distractor Gate → LLM reasoning
Hybrid (Text + Image)	Upload photo + "find me something like this but cheaper"	Both encoders → Reciprocal Rank Fusion → Cross-Encoder re-rank → LLM reasoning
Dynamic Price Filtering	"Laptops under $500"	Regex extracts price constraint → FAISS overfetch → numeric post-filter
Attribute Comparison	"Which of these has the most storage?"	Metadata hints injected into RAG context → LLM performs MAX/MIN reasoning
Explainable AI Badges	Product card shows "💰 Budget Pick"	LLM generates reasoning badges → backend parses JSON → frontend renders glassmorphic pills
Catalog Analytics	Statistics dashboard	`/api/statistics` → Chart.js interactive pie chart
Search History	Persistent query history panel	`localStorage`-based, per-user, survives page refresh

All use cases are served by a single /api/chat endpoint — no separate agents.

Architecture

User Input (text / image / both)
        │
        ▼
┌─────────────────────────┐
│    Intent Detection     │ ──── General chat? ──→ LLM (conversational mode)
└──────────┬──────────────┘
           │ Product query
           ▼
┌─────────────────────────┐
│  Dynamic Pre-Filter     │ ──── "under $100" → max_price = 100
└──────────┬──────────────┘
           ▼
┌─────────────────────────┐
│  Smart Router           │
│ ┌─────────┬───────────┐ │
│ │ MiniLM  │ OpenCLIP  │ │
│ │ (384D)  │ (768D)    │ │
│ └────┬────┴─────┬─────┘ │
│      │   FAISS  │       │
│      ▼          ▼       │
│ text_catalog  catalog   │
│   .index      .index    │
└──────────┬──────────────┘
           │ Late Fusion (RRF) + Price Filter
           ▼
┌─────────────────────────┐
│  Cross-Encoder Re-Rank  │ ──── ms-marco-MiniLM-L-6-v2 (top 15 → top 5)
└──────────┬──────────────┘
           ▼
┌─────────────────────────┐
│  Category Safety Net    │ ──── Out-of-domain? ──→ Llama 3.2 Vision (OOD rejection)
└──────────┬──────────────┘
           │ Valid products
           ▼
┌─────────────────────────┐
│  Llama 3.1 (Groq)       │ ──→ Generates recommendation + product IDs + XAI Badges
└──────────┬──────────────┘
           │
           ▼
┌─────────────────────────┐
│ Intent-Product Alignment│ ──→ Syncs LLM text with product gallery
└─────────────────────────┘

Technology Stack

Component	Technology	Rationale
Frontend	HTML5, CSS3, Vanilla JS	Lightweight, zero-build glassmorphism UI
Charts	Chart.js	Interactive pie chart with hover tooltips
Backend	FastAPI + Uvicorn	High-performance async API for ML inference
LLM (Reasoning)	Llama-3.1-8b-instant (Groq)	Fast, non-reasoning model — no `<think>` tag overhead
LLM (Vision Fallback)	Llama-3.2-11b-vision-preview (Groq)	Used only for OOD rejection on failed image searches
Text Embeddings	SentenceTransformers (`all-MiniLM-L6-v2`)	Ultra-fast 384D encoding for sub-10ms text search
Visual Embeddings	OpenCLIP (`ViT-B-16-SigLIP-256`)	State-of-the-art zero-shot image-text retrieval
Vector Search	FAISS (dual-index)	In-memory local vector search, no cloud DB dependency
Re-Ranking	CrossEncoder (`ms-marco-MiniLM-L-6-v2`)	Two-stage retrieve-and-rerank for precision without latency penalty
Environment	python-dotenv	Secure API key management via `.env` with Docker quote sanitization

Why these choices?

Dual-Index over Single-Index: A single OpenCLIP index added ~500ms latency for every text query. Decoupling text search to the lightweight MiniLM encoder (22.7M vs 150M params) reduced text query latency to under 10ms.
Groq over OpenAI/Anthropic: Free tier with extremely fast inference (sub-second). Ideal for a demo that needs to feel responsive.
Vanilla JS over React: Zero build step, instant deployment as static files, and the UI complexity doesn't warrant a framework.
FAISS over Pinecone/Weaviate: No external service dependencies — the entire system runs locally or on a single container.

Project Structure

AI_Agent_for_Commerce_Website/
├── backend/
│   ├── main.py                  # FastAPI app: chat endpoint, statistics API, intent detection
│   ├── search_engine.py         # CatalogSearchEngine: dual-index, RRF, Zero-Shot Gate
│   ├── requirements.txt         # Python dependencies
│   └── api_key.env              # Groq API key (gitignored)
├── frontend/
│   ├── index.html               # Main search UI with glassmorphism design
│   ├── style.css                # Global styles, animations, product cards
│   ├── script.js                # Time-aware greeting, fetch API, dynamic rendering
│   ├── statistics.html          # Category distribution dashboard
│   └── statistics.js            # Chart.js pie chart
├── src/
│   ├── data_cleaning.py     # Raw CSV → cleaned JSONL with category assignment
│   ├── download_images.py   # Download product images from Amazon URLs
│   ├── build_index.py       # Build 768D OpenCLIP FAISS index
│   └── build_text_index.py  # Build 384D MiniLM FAISS index
├── data/
│   ├── raw/                     # Original Kaggle CSV
│   └── processed/               # cleaned_catalog_with_images.jsonl, *.index files
├── documentation/
│   ├── initial_plan.md          # Original architecture proposal
│   └── updated_plan.md          # Current architecture documentation
├── .env.example                 # Template for API keys
└── .gitignore

Setup & Installation

Prerequisites

Python 3.10+
A Groq API key (free tier available)

1. Clone & Install Dependencies

git clone https://github.com/M4RC034/AI_Agent_for_Commerce_Website.git
cd AI_Agent_for_Commerce_Website
pip install -r requirements.txt

2. Set Your API Key

cp .env.example .env
# Edit .env and add your Groq API key:
# GROQ_API_KEY=gsk_your_key_here

Make sure to put you api_key.env inside the backend folder.

3. Prepare the Data (first time only)

Download the Amazon Products Sales Dataset 42K+ Items - 2025 and place the amazon_products_sales_data_uncleaned.csv file in the data/raw/ folder.

# Clean the raw Amazon CSV
python src/data_preprocess/data_cleaning.py

# Download product images
python src/data_preprocess/download_images.py

# Build the Visual FAISS index (768D, OpenCLIP)
python src/data_preprocess/build_index.py

# Build the Text FAISS index (384D, MiniLM)
python src/data_preprocess/build_text_index.py

4. Docker (How to access this interface locally)

docker build -t al-agent .
docker run -p 8000:8000 --env-file backend/api_key.env al-agent

Then go to http://localhost:8000 in your browser.

Key Design Decisions

Intent Detection Gate: Conversational queries (greetings, identity questions) bypass the retrieval pipeline entirely and go straight to the LLM. Product signals (keywords like "recommend", "laptop") override greeting patterns, so "Hi, find me a laptop" still routes to product search.
Distractor Gate over Category Matching: Instead of demanding exact category string matches between OpenCLIP labels and catalog labels (which caused false negatives), a binary distractor gate rejects clearly non-electronic images and lets FAISS return the closest visual neighbors for everything else.
Intent-Product Alignment Layer: The LLM emits hidden product IDs at the end of its response. A regex parser extracts these and filters the product array so the frontend gallery shows only items the LLM actually endorsed.
Cloud Vision Fallback (Selective): Llama 3.2 Vision is called only when local retrieval fails (zero valid products), keeping the happy path fast.
Client-Side Conversation History: Multi-turn context is maintained in the browser (no server-side storage or database needed). The frontend sends the last 5 turns as a JSON array with each request, and the backend injects them into the Groq message array. This enables follow-ups like "show me a cheaper one" while keeping the architecture stateless and deployment-simple.
Two-Stage Retrieve and Re-Rank Pipeline: To solve the inherent precision issues of Bi-encoder vector search without destroying responsiveness, the architecture uses a two-stage pattern. Phase 1 leverages FAISS and MiniLM for fast, high-recall retrieval (casting a wider net of k=15 candidates). Phase 2 uses a Cross-Encoder (ms-marco-MiniLM-L-6-v2) to perform deep semantic re-ranking, efficiently recalculating the relevance permutations of the user's query against the product titles to accurately slice the top 5. This dramatically bridges the 'Semantic Gap'. A strict Visual Safety Net prevents purely visual OpenCLIP queries from hitting the text-based Cross-Encoder, actively balancing computational latency against maximum precision.
Explainable AI (XAI) Badges: To build user trust and transparency, the system instructs the LLM to generate a short reasoning badge (e.g., ✨ Visual Match, 💰 Budget Pick, 🏆 Top Choice) for each recommended product. The backend extracts these via a robust regex parser with markdown-sanitization fallbacks (json.loads with backtick stripping), maps them to the product objects, and assigns a safe 🎯 Recommended default if the LLM omits or malforms the JSON. The frontend renders these as glassmorphic pill overlays (backdrop-filter: blur) positioned absolutely on each product card image, giving users immediate visual insight into why the AI selected each item.
Dynamic Price Pre-Filtering: Before any vector search occurs, a regex parser extracts price constraints from the user's natural language (e.g., "under $100", "cheaper than $50"). The extracted max_price is passed to the search engine, which uses an overfetch-then-filter paradigm: FAISS retrieves k×10 candidates, then _filter_and_rerank numerically eliminates items exceeding the budget and slices the true top-K. This bolts hard metadata constraints onto a pure vector engine.
Attribute Extraction Hints: To enable accurate product comparisons (e.g., "which has the most storage?"), the backend injects structured HINT lines into the RAG context string (e.g., HINT: Result 1 has 12TB Storage). This gives the 8B LLM explicit pre-extracted metadata to perform MAX/MIN reasoning without needing to parse raw product titles.
API Key Sanitization: The Groq API key loader applies .strip().strip('"').strip("'") to handle edge cases where Docker --env-file passes literal quote characters around the key value, preventing silent 401 Unauthorized failures in containerized deployments.

Known Limitations

Context Window: Conversation history is capped at 5 turns (10 messages) to stay within token limits. Very long conversations will lose early context.
Semantic Gap (Partially Mitigated): The reasoning model (Llama 3.1) cannot visually verify retrieval results. However, the Cross-Encoder re-ranking stage (Design Decision #6) now deeply re-scores the top 15 FAISS candidates against the user's actual query before they reach the LLM, dramatically reducing semantically wrong items from slipping through. The remaining gap exists for image-only queries, which bypass the text-based Cross-Encoder entirely by design.
Unweighted Late Fusion Conflicts (Partially Mitigated): When using hybrid search (image + text), the RRF algorithm still equally weights both indexes. However, the Cross-Encoder now acts as a second-pass semantic correction layer after fusion — even if a gaming mouse initially outranks keyboards in the RRF math, the Cross-Encoder will re-score ["mechanical keyboards", "Razer Gaming Mouse"] far lower than ["mechanical keyboards", "Corsair K70 Keyboard"], pushing correct items to the top. A production-grade resolution would still benefit from dynamically weighting the RRF scores based on primary intent classification.

API Documentation

FastAPI auto-generates interactive API documentation. Once the backend is running:

URL	Description
`http://127.0.0.1:8000/docs`	Swagger UI — Interactive docs where you can test all endpoints directly in the browser
`http://127.0.0.1:8000/redoc`	ReDoc — Clean, read-only API reference

Endpoints

`GET /` — Health Check

Returns server status.

{"status": "ok", "message": "Al Backend is running!"}

`GET /api/statistics` — Catalog Analytics

Returns category distribution for the statistics dashboard.

{
  "categories": {
    "Other Electronics": 5420,
    "Headphones": 3100,
    "TV & Display": 2800,
    "...": "..."
  }
}

`POST /api/chat` — The Core Agent Endpoint

The unified endpoint that handles all three use cases: general conversation, text-based recommendations, and image-based search.

Request (multipart form data):

Field	Type	Required	Description
`message`	string	Yes	User's text query
`image`	file	No	Product image (JPG, PNG, WebP, HEIC)
`history_json`	string	No	JSON array of previous conversation turns (default: `"[]"`)

Response (product recommendation):

{
  "agent_response": "According to the information you provide, these are the products...",
  "products": [
    {
      "rank": 1,
      "score": 0.95,
      "product_id": "prod_4133",
      "title": "Apple AirPods Pro 2nd Gen",
      "category": "Headphones",
      "price": "278.0",
      "image_url": "https://...",
      "url": "https://...",
      "badge": "🏆 Top Choice"
    }
  ]
}

Response (general conversation):

{
  "agent_response": "I'm Al, your AI shopping assistant! I can help you find electronics...",
  "products": []
}

Response (out-of-domain image):

{
  "agent_response": "According to the information you provided, you are likely looking for a wooden table. I'm sorry, but this appears to be outside of our catalog...",
  "products": []
}

License

This project was built as a take-home exercise by Palona AI.