My suggestion above would instead have been:

# Make near-zero weights exactly 0 to avoid numerical issues when computing
# feature importances.
sample_weight[sample_weight < epsilon] = 0

However this is just a suggestion and is open to discussion because I am not 100% sure this is the correct fix.

Setting small sample weights to 0 hurts model performance which is shown as follows, I guess the reason is those small weights could add up to be impactful at certain node, so they're not ignorable.

AUC under original implementation:

AUC under proposed fix:

AUC when setting small sample weight straight to 0:

Thanks for the report. This is kind of surprising... but so be it.

I indeed find problematic the comparison with 0.0. It is ugly and might bite us at some point :)

@MaxwellLZH could you provide the code snippet used to build the ROC curve: is it a training or testing score and if it is the testing score, do you observe the same for the training score?

Since #20443 was linked to regressor mainly, I am thinking that this is maybe related but for a classification problem now. It could be nice to check the strategy of resetting weight because setting to zeros and adding weak-learner might lead to a local optimal with a non-diverse ensemble.

Hi @glemaitre , the curve can be reproduced with the following code, where data_train.csv is the same dataset in the original issue data_train.csv

import pandas as pd
import numpy as np

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

train_data=pd.read_csv('~/Downloads/data_train.csv')
model_variables=['RH','t2m','tp_r5','swvl1','SM_r20','tp','cvh','vdi','SM_r10','SM_IDW']

X = train_data[model_variables] # Features
y = train_data.ignition_no 

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1024)

train_auc, test_auc = [], []
for n_est in [1, 3, 5, 10, 20, 50, 100]:
    est = AdaBoostClassifier(
            base_estimator=DecisionTreeClassifier(max_depth=10, random_state=0),
            n_estimators=n_est)
    est.fit(X_train, y_train)
    train_auc.append(roc_auc_score(y_train, est.predict_proba(X_train)[:, 1]))
    test_auc.append(roc_auc_score(y_test, est.predict_proba(X_test)[:, 1]))
    
train_auc

# [0.9408335858626135, 0.9843405707338944, 0.9937636343659565, 0.9972901393075558, 0.9483462433389864, 0.9472818988784156, 0.9391712786487824]

The changes I made is adding sample_weight[sample_weight < epsilon] = 0 after line 162, where epilson = np.finfo(sample_weight.dtype).eps