Préparer le repo P7 autonome : CI preprocessing, fetch-data, URLs prod.
Le dataset Sentiment140 reste hors git ; les tests TF/MLflow ne tournent pas encore sur le runner k3s.
This commit is contained in:
parent
6cd377eee7
commit
5b58da9f01
4 changed files with 67 additions and 2 deletions
30
.github/workflows/ci.yml
vendored
Normal file
30
.github/workflows/ci.yml
vendored
Normal file
|
|
@ -0,0 +1,30 @@
|
||||||
|
# CI — P7 Sentiment API
|
||||||
|
# Même fichier lu par Forgejo Actions et GitHub Actions (OC-ADR-001).
|
||||||
|
# Image job = node:22-bookworm (label runner mini-pc-k3s). Pas d'image ubuntu native.
|
||||||
|
|
||||||
|
name: CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
test-preprocessing:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Python + pytest (preprocessing only)
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
apt-get update
|
||||||
|
apt-get install -y --no-install-recommends python3 python3-pip python3-venv
|
||||||
|
python3 -m pip install --break-system-packages nltk pytest
|
||||||
|
python3 -m nltk.downloader punkt punkt_tab stopwords wordnet omw-1.4
|
||||||
|
python3 -m pytest deployment/api/tests/test_preprocessing.py -q
|
||||||
|
|
||||||
|
# test_main.py charge TensorFlow + artefacts MLflow (prod). Pas dans ce job :
|
||||||
|
# trop lourd pour le runner (DinD 8 Gio) et exige des secrets MLflow.
|
||||||
|
# Build/push image : Phase 6 (dual registry), après secrets Forgejo.
|
||||||
8
.gitignore
vendored
8
.gitignore
vendored
|
|
@ -1,2 +1,10 @@
|
||||||
**/mlruns
|
**/mlruns
|
||||||
**/downloaded_artifacts
|
**/downloaded_artifacts
|
||||||
|
**/.env
|
||||||
|
**/.env.*
|
||||||
|
**/.pytest_cache
|
||||||
|
**/__pycache__
|
||||||
|
*.pem
|
||||||
|
*aws*
|
||||||
|
sentiment-api-*.json
|
||||||
|
|
||||||
|
|
|
||||||
14
README.md
14
README.md
|
|
@ -4,6 +4,14 @@
|
||||||
|
|
||||||
This repository contains the notebooks and source code for the sentiment analysis project on tweets, conducted as part of the OpenClassrooms **"Air Paradis"** program.
|
This repository contains the notebooks and source code for the sentiment analysis project on tweets, conducted as part of the OpenClassrooms **"Air Paradis"** program.
|
||||||
|
|
||||||
|
**Source of truth** : [Forgejo](https://git.gregoiremureau.com/grgmro/oc-p07-sentiment-api) (self-hosted). GitHub, when mirrored, is the public shop window.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://git.gregoiremureau.com/grgmro/oc-p07-sentiment-api.git
|
||||||
|
# HTTPS (Cloudflare, fine for clone) :
|
||||||
|
git clone https://git.gregoiremureau.com/grgmro/oc-p07-sentiment-api.git
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Business Context
|
## Business Context
|
||||||
|
|
@ -33,7 +41,8 @@ The project aims to develop a complete sentiment analysis system, including:
|
||||||
- `deployment/`: FastAPI API code, unit tests, Dockerfile and deployment configuration,
|
- `deployment/`: FastAPI API code, unit tests, Dockerfile and deployment configuration,
|
||||||
- `docs/`: detailed documentation (blog.md), diagrams, screenshots,
|
- `docs/`: detailed documentation (blog.md), diagrams, screenshots,
|
||||||
- `presentation/`: project presentation slides (PDF),
|
- `presentation/`: project presentation slides (PDF),
|
||||||
- `../.github/workflows/`: CI/CD pipelines (automated tests, Docker build, DockerHub push).
|
- `.github/workflows/`: CI (Forgejo Actions on the homelab runner; same YAML on GitHub),
|
||||||
|
- `fetch-data.sh`: where to get Sentiment140 (dataset **not** in git).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -48,7 +57,7 @@ Training phases were primarily conducted on **Google Colab** to leverage GPU res
|
||||||
Notebooks detail the analysis and training steps.
|
Notebooks detail the analysis and training steps.
|
||||||
The API is available in the `deployment/api` folder with Streamlit testing interface.
|
The API is available in the `deployment/api` folder with Streamlit testing interface.
|
||||||
|
|
||||||
To test the production API: https://sentiment-api.greg-madman-nas.duckdns.org
|
To test the production API: https://api.gregoiremureau.com/ml/sentiment/
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -56,6 +65,7 @@ To test the production API: https://sentiment-api.greg-madman-nas.duckdns.org
|
||||||
|
|
||||||
- The `requirements.txt` files in `notebooks/` come from Google Colab environments and may contain dependencies not optimized for local use.
|
- The `requirements.txt` files in `notebooks/` come from Google Colab environments and may contain dependencies not optimized for local use.
|
||||||
- The `.env` file, excluded from the repository, contains necessary environment variables (MLFlow, MinIO, alerting configuration).
|
- The `.env` file, excluded from the repository, contains necessary environment variables (MLFlow, MinIO, alerting configuration).
|
||||||
|
- Training data: run `./fetch-data.sh` (Sentiment140 is **not** stored in git).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
17
fetch-data.sh
Executable file
17
fetch-data.sh
Executable file
|
|
@ -0,0 +1,17 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Datasets hors git (OC-ADR-003). Ce script n'en télécharge pas : il documente la source.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
cat <<'EOF'
|
||||||
|
P7 — Sentiment140 (tweets annotés positif/négatif)
|
||||||
|
|
||||||
|
Source amont :
|
||||||
|
http://help.sentiment140.com/for-students
|
||||||
|
archive historique : http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip
|
||||||
|
miroir Kaggle : https://www.kaggle.com/datasets/kazanova/sentiment140
|
||||||
|
|
||||||
|
Ne pas versionner le CSV/zip dans ce repo.
|
||||||
|
|
||||||
|
Copie locale éventuelle : archive scolaire Nextcloud
|
||||||
|
Master_openclassroom_AI_Engineer/07 Réalisez une analyse de sentiments...
|
||||||
|
EOF
|
||||||
Loading…
Add table
Reference in a new issue