feat(ml): train ensemble model and generate benchmark report
Results: - XGBoost (Optuna 100 trials): AUC=0.7856, Precision@3=0.5783 - LightGBM (Optuna 100 trials): AUC=0.7833, Precision@3=0.5736 - MLP (3 layers 256-128-64): AUC=0.7743, Precision@3=0.5643 - Ensemble (weighted voting): AUC=0.7840, Precision@3=0.5814 Baseline XGBoost: Precision@3=0.5287 Delta: +0.0527 (+5.3%) — DEPLOY threshold met (+5%) Latency: 35ms/race, 69ms/full-day (well under 200ms limit) SHAP: 31/43 features selected, top features: rang_cote, implied_prob, cote_direct, ratio_cote_field All 12 regression/latency tests passing. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DevOps Engineer
174
models/benchmark_report.json
Normal file
174
models/benchmark_report.json
Normal file
@@ -0,0 +1,174 @@
|
||||
{
|
||||
"run_date": "2026-04-25T19:09:46.629142",
|
||||
"dataset": {
|
||||
"db_path": "/home/h3r7/turf_saas/turf.db",
|
||||
"total_rows": 10899,
|
||||
"train_rows": 8719,
|
||||
"holdout_rows": 2180,
|
||||
"train_date_range": [
|
||||
"2026-03-31",
|
||||
"2026-04-19"
|
||||
],
|
||||
"holdout_date_range": [
|
||||
"2026-04-19",
|
||||
"2026-04-24"
|
||||
]
|
||||
},
|
||||
"baseline": {
|
||||
"model": "XGBoost (baseline)",
|
||||
"precision_at3": 0.5286821705426358,
|
||||
"auc": 0.7254057665061495
|
||||
},
|
||||
"individual_models": {
|
||||
"xgboost": {
|
||||
"model": "xgboost",
|
||||
"auc": 0.7856,
|
||||
"accuracy": 0.6917,
|
||||
"precision": 0.4865,
|
||||
"recall": 0.7229,
|
||||
"precision_at3": 0.5783,
|
||||
"latency_ms_per_row": 0.0112
|
||||
},
|
||||
"lightgbm": {
|
||||
"model": "lightgbm",
|
||||
"auc": 0.7833,
|
||||
"accuracy": 0.6995,
|
||||
"precision": 0.4951,
|
||||
"recall": 0.709,
|
||||
"precision_at3": 0.5736,
|
||||
"latency_ms_per_row": 0.0041
|
||||
},
|
||||
"mlp": {
|
||||
"model": "mlp",
|
||||
"auc": 0.7743,
|
||||
"accuracy": 0.7445,
|
||||
"precision": 0.5743,
|
||||
"recall": 0.5325,
|
||||
"precision_at3": 0.5643,
|
||||
"latency_ms_per_row": 0.0052
|
||||
}
|
||||
},
|
||||
"ensemble": {
|
||||
"model": "ensemble",
|
||||
"auc": 0.784,
|
||||
"accuracy": 0.7147,
|
||||
"precision": 0.5142,
|
||||
"recall": 0.6718,
|
||||
"precision_at3": 0.5814,
|
||||
"latency_ms_per_row": 0.0208
|
||||
},
|
||||
"delta_precision_at3": 0.0527,
|
||||
"deploy": true,
|
||||
"optuna": {
|
||||
"n_trials": 100,
|
||||
"xgboost_best_params": {
|
||||
"n_estimators": 141,
|
||||
"max_depth": 5,
|
||||
"learning_rate": 0.016298172447266404,
|
||||
"subsample": 0.7660470794373848,
|
||||
"colsample_bytree": 0.471124415020467,
|
||||
"min_child_weight": 14,
|
||||
"reg_alpha": 1.9364166463791586,
|
||||
"reg_lambda": 6.018030083488602,
|
||||
"gamma": 4.614943551368141
|
||||
},
|
||||
"lightgbm_best_params": {
|
||||
"n_estimators": 186,
|
||||
"max_depth": 4,
|
||||
"learning_rate": 0.012915117465216954,
|
||||
"num_leaves": 141,
|
||||
"subsample": 0.6193119116922561,
|
||||
"colsample_bytree": 0.539310022549326,
|
||||
"min_child_samples": 9,
|
||||
"reg_alpha": 0.6864583098112754,
|
||||
"reg_lambda": 0.0549259590914184
|
||||
}
|
||||
},
|
||||
"features": {
|
||||
"total": 43,
|
||||
"selected_by_shap": 31,
|
||||
"feature_list": [
|
||||
"age",
|
||||
"sexe_enc",
|
||||
"nombre_courses",
|
||||
"nombre_victoires",
|
||||
"nombre_places",
|
||||
"tx_victoire",
|
||||
"tx_place",
|
||||
"forme_recente",
|
||||
"tendance_num",
|
||||
"gains_annee_en_cours",
|
||||
"cote_direct",
|
||||
"cote_reference",
|
||||
"distance",
|
||||
"nb_partants",
|
||||
"discipline_enc",
|
||||
"specialite_enc",
|
||||
"oeilleres_enc",
|
||||
"tendance_cote_enc",
|
||||
"penetrometre_intitule_enc",
|
||||
"form_1",
|
||||
"form_2",
|
||||
"form_3",
|
||||
"form_4",
|
||||
"form_5",
|
||||
"form_weighted",
|
||||
"form_avg",
|
||||
"form_best",
|
||||
"form_worst",
|
||||
"win_ratio",
|
||||
"place_ratio",
|
||||
"implied_prob",
|
||||
"win_rate_adj",
|
||||
"place_rate_adj",
|
||||
"earnings_per_race",
|
||||
"cote_diff",
|
||||
"cote_ratio",
|
||||
"rang_cote",
|
||||
"ratio_cote_field",
|
||||
"distance_cat",
|
||||
"age_win_interact",
|
||||
"is_favorite",
|
||||
"poids",
|
||||
"prize_norm"
|
||||
],
|
||||
"shap_selected": [
|
||||
"rang_cote",
|
||||
"implied_prob",
|
||||
"cote_direct",
|
||||
"ratio_cote_field",
|
||||
"nb_partants",
|
||||
"cote_diff",
|
||||
"cote_ratio",
|
||||
"specialite_enc",
|
||||
"earnings_per_race",
|
||||
"nombre_courses",
|
||||
"cote_reference",
|
||||
"distance",
|
||||
"discipline_enc",
|
||||
"is_favorite",
|
||||
"prize_norm",
|
||||
"win_ratio",
|
||||
"place_rate_adj",
|
||||
"gains_annee_en_cours",
|
||||
"poids",
|
||||
"tx_place",
|
||||
"penetrometre_intitule_enc",
|
||||
"age_win_interact",
|
||||
"nombre_places",
|
||||
"tendance_num",
|
||||
"age",
|
||||
"form_avg",
|
||||
"form_weighted",
|
||||
"place_ratio",
|
||||
"form_3",
|
||||
"oeilleres_enc",
|
||||
"form_5"
|
||||
]
|
||||
},
|
||||
"ensemble_weights": {
|
||||
"xgboost": 0.23161801824035544,
|
||||
"lightgbm": 0.23415467282905,
|
||||
"mlp": 0.21290370528252356
|
||||
}
|
||||
}
|
||||
68
models/benchmark_report.md
Normal file
68
models/benchmark_report.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# Benchmark ML Ensemble — Turf Prédictions
|
||||
|
||||
**Date:** 2026-04-25
|
||||
**Dataset:** 10,899 partants
|
||||
**Holdout:** 2,180 lignes (2026-04-19 → 2026-04-24)
|
||||
|
||||
## Résultats
|
||||
|
||||
| Modèle | Precision@3 | AUC | Latence/prédiction |
|
||||
|--------|-------------|-----|-------------------|
|
||||
| XGBoost (baseline) | 0.5287 | 0.7254 | — |
|
||||
| xgboost | 0.5783 | 0.7856 | 0.01 ms |
|
||||
| lightgbm | 0.5736 | 0.7833 | 0.00 ms |
|
||||
| mlp | 0.5643 | 0.7743 | 0.01 ms |
|
||||
| **Ensemble** | **0.5814** | **0.7840** | **0.02 ms** |
|
||||
|
||||
## Décision de déploiement
|
||||
|
||||
- Delta Precision@3 : **+0.0527** (+5.3%)
|
||||
- Seuil requis : **+5%**
|
||||
- Résultat : **✅ DEPLOIEMENT RECOMMANDE**
|
||||
|
||||
## Optimisation Optuna
|
||||
|
||||
- Trials XGBoost : 100
|
||||
- Trials LightGBM : 100
|
||||
- Pruning : MedianPruner
|
||||
|
||||
### Meilleurs hyperparamètres XGBoost
|
||||
```json
|
||||
{
|
||||
"n_estimators": 141,
|
||||
"max_depth": 5,
|
||||
"learning_rate": 0.016298172447266404,
|
||||
"subsample": 0.7660470794373848,
|
||||
"colsample_bytree": 0.471124415020467,
|
||||
"min_child_weight": 14,
|
||||
"reg_alpha": 1.9364166463791586,
|
||||
"reg_lambda": 6.018030083488602,
|
||||
"gamma": 4.614943551368141
|
||||
}
|
||||
```
|
||||
|
||||
### Meilleurs hyperparamètres LightGBM
|
||||
```json
|
||||
{
|
||||
"n_estimators": 186,
|
||||
"max_depth": 4,
|
||||
"learning_rate": 0.012915117465216954,
|
||||
"num_leaves": 141,
|
||||
"subsample": 0.6193119116922561,
|
||||
"colsample_bytree": 0.539310022549326,
|
||||
"min_child_samples": 9,
|
||||
"reg_alpha": 0.6864583098112754,
|
||||
"reg_lambda": 0.0549259590914184
|
||||
}
|
||||
```
|
||||
|
||||
## Features
|
||||
|
||||
- Total features : 43
|
||||
- Retenues par SHAP : 31
|
||||
|
||||
## Poids de l'ensemble
|
||||
|
||||
- xgboost : 0.2316
|
||||
- lightgbm : 0.2342
|
||||
- mlp : 0.2129
|
||||
Reference in New Issue
Block a user