Attention-Based Dimensionality Reduction for Feature Selection

Attention-Based Feature Dimensionality Reduction

Overview

This project explores novel dimensionality reduction techniques that incorporate attention mechanisms — specifically PCA + Attention and t-SNE + Attention. The work spanned roughly 40 days (August 21 – September 30, 2023), across 42 commits, and was carried out as an undergraduate research project.

Motivation

Dimensionality reduction — transforming high-dimensional data into a lower-dimensional representation — is a foundational preprocessing step in machine learning. Conventional methods such as PCA and t-SNE share a key limitation: they treat all features equally. This project investigates whether using an attention mechanism to assign higher weight to more informative features can improve the quality of the resulting low-dimensional representation.

Research Process

Phase 1: Baseline Implementation (08/21–08/28)

Completed baseline implementations of PCA and t-SNE, and designed the attention module.

import numpy as np
import torch
import torch.nn as nn

class AttentionFeatureSelector(nn.Module):
    """Attention module that learns per-feature importance weights"""
    
    def __init__(self, input_dim):
        super().__init__()
        self.attention = nn.Sequential(
            nn.Linear(input_dim, input_dim // 2),
            nn.ReLU(),
            nn.Linear(input_dim // 2, input_dim),
            nn.Softmax(dim=-1)
        )
    
    def forward(self, x):
        weights = self.attention(x)  # per-feature importance
        return x * weights  # apply weights

Phase 2: Integrated Module Development (08/28–09/15)

As reflected in the commit message 인테그 모듈추가, this phase focused on building a unified pipeline.

class AttentionPCA:
    """Unified Attention + PCA pipeline"""
    
    def __init__(self, n_components):
        self.attention = AttentionFeatureSelector(input_dim)
        self.pca = PCA(n_components=n_components)
    
    def fit_transform(self, X):
        # Step 1: learn feature weights via attention
        X_weighted = self.attention(torch.tensor(X))
        
        # Step 2: apply PCA to the weighted data
        X_reduced = self.pca.fit_transform(X_weighted.detach().numpy())
        return X_reduced

Phase 3: numpy → tensor Migration (09/01–09/15)

Operations originally written in numpy were ported to PyTorch tensors. The commit message numpy -> tensor process 필요!!! captures the urgency at the time.

# Before (numpy)
attention_weights = np.dot(X, W) 
X_weighted = X * attention_weights

# After (PyTorch tensor - GPU 활용 가능)
attention_weights = torch.matmul(X_tensor, W_tensor)
X_weighted = X_tensor * attention_weights

Phase 4: GPU Troubleshooting (09/15–09/25)

The commit message FuxkGPU (!) says it all — GPU training wasn't behaving as expected and required significant debugging.

Key issues:

• CUDA out-of-memory errors requiring batch size reduction
• CPU/GPU tensor device mismatches (missing .to(device) calls)
• Compatibility issues between gradient computation and PCA

Phase 5: Collaborative Experiments and Final Results (09/25–09/30)

Experiments were extended through collaboration on the jinsu branch, and final results were consolidated in the 0930_final commit.

Experimental Results

Comparative experiments across multiple datasets:

The attention-augmented methods achieved 1–2 percentage point improvements in classification accuracy over their standard counterparts.

Tech Stack

• Python 3.x, PyTorch (model implementation)
• scikit-learn (PCA, t-SNE, evaluation)
• NumPy → PyTorch tensor (migration)
• matplotlib (visualization)
• CUDA (GPU training)