oc-p11-bigdata-fruits/README.md

Big Data Cloud Processing Project - Fruits!

📘 Ce projet est également disponible en français 🇫🇷

Source of truth : https://git.gregoiremureau.com/openclassroom-ai/oc-p11-bigdata-fruits

git clone https://git.gregoiremureau.com/openclassroom-ai/oc-p11-bigdata-fruits.git

Dataset: see fetch-data.sh (excluded from git if large).

📋 Project Context

This project was carried out as part of a consulting assignment for Fruits!, a young AgriTech start-up developing innovative solutions for fruit harvesting. The company aims to preserve fruit biodiversity by enabling species-specific processing through the development of intelligent harvesting robots.

🎯 Objectives

Complete a feature extraction processing pipeline in a Big Data environment on the AWS cloud:

1. Resuming the work : Take over the incomplete notebook left by a previous apprentice
2. Completing the pipeline : Implement the missing parts (weight broadcasting, distributed PCA)
3. Production deployment : Deploy the solution on a GDPR-compliant AWS EMR cluster
4. Cost optimization : Keep execution costs under €10

The long-term objective is to build a fruit image classification engine for a mobile app raising public awareness.

📊 Data

The dataset used is Fruits-360, available on Kaggle:

• 147,691 images of fruits of different varieties
• Format: 100x100 pixels, color
• Specific test set: 103 images of multiple fruits
• ⚠️ Important note : The dataset is not included in this repository due to its size
• URL : https://www.kaggle.com/moltean/fruits

📁 Project Structure

├── P11_01_notebook.ipynb
├── P11_02_images/
│   ├── Results_PCA/                    # PCA results in Parquet format
│   │   ├── part-00000-*.parquet
│   │   ├── part-00001-*.parquet
│   │   └── ...
│   └── test-multiple-fruits/           # Test images (103 images)
│       ├── apple.jpg
│       ├── apples1.jpg
│       └── ...
└── P11_03_présentation.pdf

1. P11_01_notebook.ipynb

Main PySpark Big Data processing notebook

This notebook contains the complete processing pipeline:

• Initialization : SparkSession configuration and S3 connection
• Data loading : Reading images from S3 in binary format
• Transfer Learning : Feature extraction via MobileNetV2 pre-trained on ImageNet
  • Removal of the final classification layer
  • Extraction of 1280-dimensional vectors
  • Broadcasting the model weights to the workers (critical addition)
• Dimensionality reduction : Distributed PCA with Spark ML
  • Reduction from 1280 → 70 dimensions
  • 99.94% of variance retained
  • Storage and performance optimization
• Results export : Saving in Parquet format on S3

Key optimizations implemented :

• Pandas UDF Scalar Iterator for batch processing
• Broadcasting the model weights to avoid repeated loading
• Distributed PCA with parallelized computation on the cluster

2. P11_02_images/

Images and results folder

• Results_PCA/ : Dimensionality reduction results stored in distributed Parquet format (20 files)
• test-multiple-fruits/ : 103 varied test images used to validate the pipeline

3. P11_03_présentation.pdf

Results presentation

This presentation summarizes:

• The context and challenges of the project
• The cloud architecture put in place (AWS EMR, S3, EC2)
• The cluster configuration and Spark optimizations
• The complete algorithmic pipeline
• The technical results obtained
• The improvement and deployment perspectives

🛠️ Technologies Used

Cloud & Infrastructure

• AWS EMR : Distributed computing cluster (Spark, Hadoop)
• AWS S3 : Storage of data and results
• AWS EC2 : m5.xlarge instances (1 Master + 2 Workers)
• EU-West-3 Region (Paris) : GDPR compliance

Big Data & Processing

• PySpark : Distributed computing framework
• Hadoop : Distributed file system
• Spark ML : Distributed Machine Learning library

Machine Learning

• TensorFlow : Deep learning framework
• MobileNetV2 : Transfer learning model pre-trained on ImageNet
• PCA : Dimensionality reduction algorithm

Development Tools

• JupyterHub : Development environment on the cluster
• SSH Tunneling : Secure access to the cluster
• Python 3 : Programming language

📈 Technical Architecture

EMR Cluster Configuration

Instances :

• 1 Master node (Spark driver) - m5.xlarge
• 2 Worker nodes (Spark executors) - m5.xlarge
• Region: eu-west-3 (Paris)

Installed software :

• Hadoop 3.2.1
• Spark 3.1.2
• JupyterHub 1.4.1
• TensorFlow 2.4.1

Bootstrap : Automatic installation of Python packages

sudo python3 -m pip install numpy pandas pillow pyarrow fsspec s3fs

Secure Access

SSH Tunneling with Port Forwarding :

ssh -i ./emr-keypair.pem -L 8890:localhost:8890 -L 9443:localhost:9443 hadoop@[IP-EMR]

• Port 9443: JupyterHub (development)
• Port 8890: Spark monitoring interface

Advantages :

• End-to-end encrypted connection
• No external proxy required
• Precise access control

Processing Pipeline

1. Image loading : Binary format from S3, label extraction from paths
2. Transfer Learning : MobileNetV2 → 1280-dimension features
   • Broadcasting weights for efficient distribution
   • Pandas UDF Scalar Iterator for batch processing
3. Distributed PCA : Reduction from 1280 → 70 dimensions
   • Distributed computation of the covariance matrix
   • Extraction of principal components
   • Feature transformation
4. Export : Saving in Parquet format on S3

📝 Methodology

1. Understanding : Analysis of the existing notebook and identification of gaps
2. Local development : Testing and validation on a reduced sample
3. Cloud migration : Deployment on the EMR cluster
4. Optimization : Adding weight broadcasting and distributed PCA
5. Validation : Testing on 103 images, scalability verification

💰 Cost Management

Objective : < €10 for validation

Optimization strategies :

• Use of a modest cluster (1 Master + 2 Workers)
• m5.xlarge instances (performance/cost trade-off)
• Testing on a reduced sample before scaling
• Shutting down the cluster after use
• Local server for development and testing

Result : Validation cost < €5 (less than 1h of cluster time)

🎯 Results

Technical

• ✅ Dimensionality reduction: 1280 → 70 dimensions
• ✅ Variance retained: 99.94%
• ✅ Compression factor: x4.1
• ✅ Complete and functional pipeline
• ✅ Validated scalable architecture

Compliance

• ✅ GDPR : EU-West-3 Region (Paris)
• ✅ Security : SSH access, secure tunneling
• ✅ Costs : < €10 for validation

Implemented features

• ✅ MobileNetV2 weight broadcasting (missing in the initial version)
• ✅ Distributed PCA with Spark ML (missing in the initial version)
• ✅ Memory optimization with Pandas UDF Scalar Iterator
• ✅ Results export in Parquet format

🚀 Outlook

Short term

PCA optimization :

• 70 components valid for 103 test images
• With the full dataset (~150k images), adjust the number of components k
• Test different values to optimize variance/dimensions

Final classification :

• Train a classification model on the PCA features
• Evaluate performance on the test set
• Fine-tuning the number of components

Medium term

Production deployment :

• Integration of the model into the mobile application
• Automated CI/CD pipeline
• Real-time performance monitoring
• Model version management

Long term

Future extensions :

• Using the application as a marketing and technical MVP
• Collecting new field data
• Continuous model improvement
• Integration with intelligent harvesting robots

🎓 Skills Developed

• Big Data architecture on the cloud (AWS EMR, S3, EC2)
• Distributed processing with PySpark
• Transfer Learning and feature extraction
• Large-scale dimensionality reduction
• Spark performance optimization
• Cloud cost management
• GDPR compliance for data storage

👤 Author

Grégoire Mureau
Completion date: October 2025

---

This project was carried out as part of the AI Engineer program