Street Vision PoC - Urban Semantic Segmentation

📘 This project is also available in French 🇫🇷

Source of truth : Forgejo (self-hosted).

git clone https://git.gregoiremureau.com/openclassroom-ai/oc-p09-street-vision-poc.git
git clone https://git.gregoiremureau.com/openclassroom-ai/oc-p09-street-vision-poc.git

Prod: https://api.gregoiremureau.com/ml/street-vision-poc/

Dataset: ./fetch-data.sh (Cityscapes not in git).

🎯 Project Description

Street Vision PoC is a proof of concept demonstrating the improvement of urban semantic segmentation performance by comparing traditional CNN architectures (FPN) with modern Transformer approaches (SegFormer).

This project empirically evaluates the potential of Vision Transformers for understanding complex urban scenes, with a focus on safety-critical classes (pedestrians, vehicles, signage).

🔬 Technical Context

• Dataset: Cityscapes (2,975 high-resolution pixel-by-pixel annotated images)
• Compared architectures:
  • Traditional CNN: FPN + EfficientNet-B0/ResNet34
  • Vision Transformer: SegFormer-B0/B1
• Evaluation resolutions: 512×512 and 768×768 pixels
• Main metric: Mean Intersection over Union (mIoU)

📊 Main Results

Model	mIoU	Parameters	Size	Performance
SegFormer-B1	79.98%	13.68M	52MB	⭐ Best
FPN+ResNet34	79.22%	23.16M	88MB	🔄 Strong baseline
FPN+EfficientNet-B0	78.67%	5.76M	22MB	💡 Compact
SegFormer-B0	77.91%	3.72M	14MB	⚡ Ultra-lightweight

Comparative analysis: SegFormer-B1 and FPN+ResNet34 achieve similar performance (~79-80% mIoU), but SegFormer-B1 has the advantage of being more compact and likely faster on GPU. The SegFormer-B2 to B5 versions would promise significant gains, but my hardware constraints did not allow their evaluation.

🎯 Significant Gains by Critical Classes

• Human: +1.78% IoU (safety-critical)
• Vehicle: +1.05% IoU (mobile obstacles)
• Object: +0.56% IoU (urban signage)

🚀 Quick Deployment

Docker Hub

The application is containerized and available on Docker Hub:

docker pull grgmdmn/street-vision-poc_api
# live prod:
# https://api.gregoiremureau.com/ml/street-vision-poc/

Configuration with Docker Compose

1. Copy the example configuration file:

cp docker-compose.yml.example docker-compose.yml

2. Configure the environment variables in docker-compose.yml.

3. Launch the application:

docker-compose up -d

4. Access the interface: http://localhost:$(port)

🎮 Interface Features

Interactive Dashboard

• Image upload: Drag-and-drop of urban images
• Dual inference: Simultaneous comparison of FPN vs SegFormer
• Visualization: Segmentation masks colored by class
• Real-time metrics: Inference time and confidence scores
• Export: Download of segmentation results
• WCAG accessibility: Interface compliant with accessibility standards

Exploratory Data Analysis

• Cityscapes statistics: Class distribution, native resolutions
• Interactive charts: Pixel breakdown by meta-class
• Annotated examples: Visualization of dataset samples

📈 Evaluation Methodology

Comparative Approach

1. CNN baseline: FPN+EfficientNet/ResNet retraining at higher resolutions
2. New approach: SegFormer evaluation with global attention
3. Rigorous protocol: Cross-validation, standardized metrics
4. Per-class analysis: Detailed performance across 8 urban meta-classes

Identified Limitations

• Fixed resolution: Evaluation without sliding window (vs. original protocol)
• Limited resources: Tests only on SegFormer-B0/B1
• Controlled conditions: Cityscapes dataset only (favorable weather)

🔬 Technical Insights

SegFormer Advantages

• Global attention: Every pixel "sees" the entire image simultaneously
• Lighter architecture: Compact model with fewer parameters
• Contextual robustness: Better handling of occlusions and complex objects
• Improvement potential: B2-B5 versions would promise significant gains

Identified Trade-offs

• Classic CPU speed: ~2x slower than FPN on high-performance processors (Xeon, 9600X)
• Energy-efficient CPU speed: 3x faster on energy-efficient processors (N100)
• GPU optimization: Transformer architecture favored by hardware acceleration

📚 References

• Xie, E. et al. (2021). SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. NeurIPS 2021.
• Cordts, M. et al. (2016). The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR 2016.
• Segmentation Models PyTorch - Implementation framework used

---

Note: This proof of concept demonstrates the potential of Transformer architectures for urban segmentation, while highlighting the importance of computational resources to fully leverage their capabilities.