# Street Vision PoC - Urban Semantic Segmentation

📘 Ce projet est également disponible en [français 🇫🇷](./README.fr.md)

**Source of truth** : [Forgejo](https://git.gregoiremureau.com/openclassroom-ai/oc-p09-street-vision-poc) (self-hosted).

```bash
git clone https://git.gregoiremureau.com/openclassroom-ai/oc-p09-street-vision-poc.git
```

Prod: https://api.gregoiremureau.com/ml/street-vision-poc/

Dataset: `./fetch-data.sh` (Cityscapes **not** in git).

## 🎯 Project Description

**Street Vision PoC** is a proof of concept demonstrating the improvement of urban semantic segmentation performance by comparing traditional CNN architectures (FPN) with modern Transformer approaches (SegFormer).

This project empirically evaluates the potential of **Vision Transformers** for understanding complex urban scenes, with a focus on safety-critical classes (pedestrians, vehicles, signage).

### 🔬 Technical Context

- **Dataset**: Cityscapes (2,975 high-resolution pixel-by-pixel annotated images)
- **Compared architectures**:
  - **Traditional CNN**: FPN + EfficientNet-B0/ResNet34
  - **Vision Transformer**: SegFormer-B0/B1
- **Evaluation resolutions**: 512×512 and 768×768 pixels
- **Main metric**: Mean Intersection over Union (mIoU)

## 📊 Main Results

| **Model** | **mIoU** | **Parameters** | **Size** | **Performance** |
|------------|----------|----------------|------------|-----------------|
| **SegFormer-B1** | **79.98%** | 13.68M | 52MB | ⭐ **Best** |
| **FPN+ResNet34** | 79.22% | 23.16M | 88MB | 🔄 Strong baseline |
| **FPN+EfficientNet-B0** | 78.67% | 5.76M | 22MB | 💡 Compact |
| **SegFormer-B0** | 77.91% | 3.72M | 14MB | ⚡ Ultra-lightweight |

**Comparative analysis**: SegFormer-B1 and FPN+ResNet34 achieve similar performance (~79-80% mIoU), but SegFormer-B1 has the advantage of being more compact and likely faster on GPU. The SegFormer-B2 to B5 versions would promise significant gains, but my hardware constraints did not allow their evaluation.

### 🎯 Significant Gains by Critical Classes

- **Human**: +1.78% IoU (safety-critical)
- **Vehicle**: +1.05% IoU (mobile obstacles)
- **Object**: +0.56% IoU (urban signage)

## 🚀 Quick Deployment

### Docker Hub

The application is containerized and available on Docker Hub:

```bash
docker pull grgmdmn/street-vision-poc_api
# live prod:
# https://api.gregoiremureau.com/ml/street-vision-poc/
```

### Configuration with Docker Compose

1. Copy the example configuration file:
```bash
cp docker-compose.yml.example docker-compose.yml
```

2. Configure the environment variables in `docker-compose.yml`.

3. Launch the application:
```bash
docker-compose up -d
```

4. Access the interface: `http://localhost:$(port)`

## 🎮 Interface Features

### Interactive Dashboard

- **Image upload**: Drag-and-drop of urban images
- **Dual inference**: Simultaneous comparison of FPN vs SegFormer
- **Visualization**: Segmentation masks colored by class
- **Real-time metrics**: Inference time and confidence scores
- **Export**: Download of segmentation results
- **WCAG accessibility**: Interface compliant with accessibility standards

### Exploratory Data Analysis

- **Cityscapes statistics**: Class distribution, native resolutions
- **Interactive charts**: Pixel breakdown by meta-class
- **Annotated examples**: Visualization of dataset samples

## 📈 Evaluation Methodology

### Comparative Approach

1. **CNN baseline**: FPN+EfficientNet/ResNet retraining at higher resolutions
2. **New approach**: SegFormer evaluation with global attention
3. **Rigorous protocol**: Cross-validation, standardized metrics
4. **Per-class analysis**: Detailed performance across 8 urban meta-classes

### Identified Limitations

- **Fixed resolution**: Evaluation without sliding window (vs. original protocol)
- **Limited resources**: Tests only on SegFormer-B0/B1
- **Controlled conditions**: Cityscapes dataset only (favorable weather)

## 🔬 Technical Insights

### SegFormer Advantages

- **Global attention**: Every pixel "sees" the entire image simultaneously
- **Lighter architecture**: Compact model with fewer parameters
- **Contextual robustness**: Better handling of occlusions and complex objects
- **Improvement potential**: B2-B5 versions would promise significant gains

### Identified Trade-offs

- **Classic CPU speed**: ~2x slower than FPN on high-performance processors (Xeon, 9600X)
- **Energy-efficient CPU speed**: 3x faster on energy-efficient processors (N100)
- **GPU optimization**: Transformer architecture favored by hardware acceleration

## 📚 References

- **Xie, E. et al.** (2021). *SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers*. NeurIPS 2021.
- **Cordts, M. et al.** (2016). *The Cityscapes Dataset for Semantic Urban Scene Understanding*. CVPR 2016.
- **Segmentation Models PyTorch** - [Implementation framework used](https://github.com/qubvel/segmentation_models.pytorch)

---

**Note**: This proof of concept demonstrates the potential of Transformer architectures for urban segmentation, while highlighting the importance of computational resources to fully leverage their capabilities.