Source of truth : https://git.gregoiremureau.com/openclassroom-ai/oc-p03-openfoodfacts

git clone https://git.gregoiremureau.com/openclassroom-ai/oc-p03-openfoodfacts.git
git clone https://git.gregoiremureau.com/openclassroom-ai/oc-p03-openfoodfacts.git

Dataset: see fetch-data.sh (outside of git if large).

Data Preparation Project - OpenFoodFacts

📘 Ce projet est également disponible en français 🇫🇷

📋 Project Context

This project was carried out for a public health organization wishing to improve the OpenFoodFacts collaborative database. The objective is to prepare and analyze food data to facilitate the creation of a product entry assistance system for users, while respecting GDPR principles.

OpenFoodFacts is an open database of food products from around the world, containing nutritional information, ingredients, allergens and labels. However, this data requires thorough cleaning and statistical exploration to be usable.

🎯 Objectives

The main objectives of the project are:

1. Clean and filter the data: Eliminate duplicates, inconsistencies, and select relevant variables
2. Handle missing and outlier values: Identify and manage missing data, estimate nutritional values
3. Perform statistical analyses: Univariate, bivariate and multivariate analyses to understand relationships between variables
4. Create new features: Feature engineering to enrich the dataset
5. Produce visualizations: Charts and analyses to facilitate data understanding
6. Comply with GDPR: Ensure compliance with data protection regulations

📊 Data

The data comes from the OpenFoodFacts database, a global collaborative database on food products containing:

• Nutritional information (energy, lipids, carbohydrates, proteins, etc.)
• Lists of ingredients and allergens
• Nutritional scores (Nutri-Score)
• Labels and certifications
• Geographic and packaging information

⚠️ Important note: The original dataset is large and is not included in this repository. It can be downloaded at the following address:

• URL: https://s3-eu-west-1.amazonaws.com/static.oc-static.com/prod/courses/files/parcours-data-scientist/P2/fr.openfoodfacts.org.products.csv.zip

Additional note: The NutriScoreCalculator library was used to complete certain nutritional values and calculate the missing Nutri-Score.

📁 Project Structure

├── openfoodfacts_analysis_clean.ipynb
├── OpenFoodFacts_presentation.pdf
├── data_analysis_functions.py
├── pandas_df_management.py
├── strings_management.py
├── translated_allergens.json
└── translated_ingredients.json

1. openfoodfacts_analysis_clean.ipynb

Main Data Analysis and Preparation Notebook

This notebook is the core of the project and covers the entire data cleaning and analysis process:

Loading and initial exploration:

• Importing the OpenFoodFacts dataset
• Analysis of size, data types and descriptive statistics
• Identification of relevant columns for the analysis

Cleaning and filtering:

• Selection of relevant features for the business objective
• Removal of duplicates (identical products)
• Filtering of rows without target values (notably allergens)
• Harmonization of formats and units

Handling missing and outlier values:

• Identification of missing values per variable
• Imputation strategies adapted to each variable type
• Detection and treatment of outlier nutritional values
• Estimation of missing energy_100g values via correlations
• Calculation of missing Nutri-Score with NutriScoreCalculator

Creating new features (Feature Engineering):

• allergens_count: number of allergens per product
• Features derived from ingredients (count, presence of certain ingredients)
• Binary variables for additives
• Nutritional ratios (lipids/energy, sugars/carbohydrates, etc.)

In-depth exploratory analysis:

• Univariate analyses: distributions of nutritional variables, Nutri-Score, product categories
• Bivariate analyses: correlations between numerical variables, relationships between Nutri-Score and compositions
• Multivariate analyses: PCA (Principal Component Analysis) to visualize overall relationships
• Statistical tests: Pearson correlations, ANOVA, Chi-squared tests, Cramér's V
• Visualizations: histograms, boxplots, correlation heatmaps, scatter plots

Export of refined data:

• Saving the cleaned and enriched DataFrame (df_refined_allergens_based.csv)
• Documentation of the transformations performed

2. OpenFoodFacts_presentation.pdf

Defense Presentation

This presentation summarizes:

• The context and challenges of the project
• The data cleaning and preparation methodology
• The main statistical analyses performed
• The key insights discovered in the data
• Recommendations for improving the entry system
• Prospects for improvement and exploitation of the data

3. data_analysis_functions.py

Data Analysis Functions Module

This Python module contains reusable functions for:

• Data cleaning (duplicate removal, filtering)
• Outlier identification (statistical methods, IQR)
• Outlier treatment
• Detection of inconsistencies in the data
• Generation of data quality reports

4. pandas_df_management.py

Pandas DataFrame Management Module

This module automates common DataFrame manipulation tasks:

• Functions for quick DataFrame inspection
• Automation of data type management
• Transformation and reshaping functions
• Utilities for data export and import
• Data structure validation functions

5. strings_management.py

String Cleaning and Normalization Module

This module contains specialized functions for:

• String cleaning (removal of spaces, special characters)
• Text normalization (uppercase/lowercase, accents)
• Parsing of ingredient and allergen lists
• Standardization of product and category names
• Handling of multilingual formats

6. translated_allergens.json

Allergens Mapping File

JSON file containing translations and mappings of allergens:

• Harmonization of allergen names in different languages
• Standardization of terms (gluten, lactose, nuts, etc.)
• Grouping of spelling variants
• Facilitates multilingual analysis of the OpenFoodFacts database

7. translated_ingredients.json

Ingredients Mapping File

JSON file containing translations and mappings of ingredients:

• Harmonization of common ingredient names
• Standardization of terms in different languages
• Grouping of synonyms and variants
• Support for multilingual analysis of compositions

🛠️ Technologies Used

• Python: Main language for data analysis
  • pandas, numpy: data manipulation and analysis
  • matplotlib, seaborn, plotly: data visualization
  • scipy, statsmodels: statistical tests and analyses
  • scikit-learn: PCA and multivariate analyses
  • NutriScoreCalculator: Nutri-Score calculation
• Jupyter Notebook: development and documentation
• JSON: storage of mapping files

📈 Methodology

1. Initial exploration: Understanding the structure and content of the dataset
2. Cleaning: Elimination of duplicates, inconsistencies and irrelevant data
3. Handling missing values: Imputation strategies adapted to each variable
4. Feature Engineering: Creation of relevant derived variables
5. Statistical analyses: Tests and visualizations to understand relationships
6. Validation: Verification of the quality and consistency of the cleaned data
7. Documentation: Presentation of results and recommendations

🎯 Expected Results

• ✅ Cleaned, enriched dataset ready for advanced analyses
• ✅ In-depth understanding of OpenFoodFacts data
• ✅ Identification of key variables for Nutri-Score prediction
• ✅ Insights on relationships between nutritional composition and food quality
• ✅ Reusable Python modules for similar data processing
• ✅ Complete documentation of the preparation process

💡 Key Points of the Project

Data Quality

Since OpenFoodFacts is a collaborative database, it presents many challenges:

• Significant missing values: some columns have more than 50% missing data
• Inconsistencies: variations in formats, units, languages
• Duplicates: products entered multiple times with variations
• Outlier values: entry errors in nutritional values

GDPR Compliance

The project ensures compliance with regulations:

• Anonymization of potential personal data
• Compliant processing of European data
• Documentation of the processing performed

In-depth Statistical Analyses

The project goes beyond simple cleaning by offering:

• Univariate analyses to understand distributions
• Bivariate analyses to identify relationships between variables
• Multivariate analyses (PCA) for an overall view
• Statistical tests to validate observations

Business-oriented Feature Engineering

Creation of relevant variables for public health:

• Allergen counting (helps identify at-risk products)
• Nutritional ratios (sugars/carbohydrates, lipids/energy)
• Nutritional quality indicators

🎓 Skills Developed

• Cleaning and preparation of large and imperfect datasets
• Handling missing and outlier values with adapted strategies
• Univariate, bivariate and multivariate statistical analyses
• Business-oriented feature engineering for public health
• Data visualization for communicating insights
• Development of reusable Python modules
• Compliance with regulatory constraints (GDPR)
• Documentation and presentation of technical results

👤 Author

Grégoire Mureau

---

This project illustrates the complete data preparation process in data science: from initial exploration to the refined dataset, through cleaning, enrichment and statistical analysis.