? If so, could you please try #11286 ?
\nThere's no difference for training aside from the initialization step. However, the metric calculation (pin ball) had a bug related to multi-quantiles and it's fixed in 3.0. After the fix, it should use the average of the loss across quantile targets, which may contribute to the observed difference.
\n\n\nSo basically you're saying for training with a list of alphas, there are still multiple models being trained.
\n
Yes. Similar to multi-class/label/target models.
\nIn future releases, the difference in the initialization step will be removed (so, will be completely the same). #9043 is a different path that uses a single tree for multiple targets, but it will take some time.
","upvoteCount":0,"url":"https://github.com/dmlc/xgboost/discussions/11314#discussioncomment-12419329"}}}How does the multi-alpha quantile regression training work? #11314
-
|
Hi! I am using the xgboost package (in python if this matters) to train quantile regression for multiple alphas. One can set a list of alphas in the train xgboost train function when using pin ball loss, and this would return a single model that do prediction for the quantiles at this list of alphas. I wonder how it is different from me looping over the list of alphas, and train a single xgboost model for every alpha? Could you clarify that for the single model approach (set a list of alphas in the train function and train one model), are the prediction heads for different alphas concatenated to a shared tree structure? If so, how large is the shared tree and how can I tune it? Is there any document or paper you may be able to share for this approach? Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 4 replies
-
Other than the initialization step, no difference.
No, but we are working on it. See #9043 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks so much for the feedback! So basically you're saying for training with a list of alphas, there are still multiple models being trained. We did see a slight performance drop for the single model approach than the "training with one alpha at a time" approach, so based on your feedback this is solely due to the different initialization? |
Beta Was this translation helpful? Give feedback.
-
Could you please elaborate on that? Are you suggesting that for q in (0.95, 0.5, 0.05):
xgb.train({"quantile_alphas": [q]})? If so, could you please try #11286 ? There's no difference for training aside from the initialization step. However, the metric calculation (pin ball) had a bug related to multi-quantiles and it's fixed in 3.0. After the fix, it should use the average of the loss across quantile targets, which may contribute to the observed difference.
Yes. Similar to multi-class/label/target models. In future releases, the difference in the initialization step will be removed (so, will be completely the same). #9043 is a different path that uses a single tree for multiple targets, but it will take some time. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I assume the bug fixed in 3.0 was related to this line https://github.com/dmlc/xgboost/blob/master/src/objective/quantile_obj.cu#L170, and I assume the average is over all quantiles, like mean(L_alpha) over all alphas in the list, where L_alpha is the loss function at an alpha value? |
Beta Was this translation helpful? Give feedback.
-
|
No, it's just the metric: #11279 not related to training. |
Beta Was this translation helpful? Give feedback.
Could you please elaborate on that? Are you suggesting that
xgb.train({"quantile_alphas": [0.95, 0.5, 0.05]})has lower accuracy than:? If so, could you please try #11286 ?
There's no difference for training aside from the initialization step. However, the metric calculation (pin ball) had a bug related to multi-quantiles and it's fixed in 3.0. After the fix, it should use the average of the loss across quantile targets, which may contribute to the observed difference.