The Wayback Machine - https://web.archive.org/web/20220524192642/https://github.com/scikit-learn/scikit-learn/pull/19415
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH Add mean_pinball_loss metric for quantile regression #19415

Merged
merged 63 commits into from Feb 18, 2021
Merged

ENH Add mean_pinball_loss metric for quantile regression #19415

merged 63 commits into from Feb 18, 2021

Conversation

sdpython
Copy link
Contributor

@sdpython sdpython commented Feb 9, 2021

Reference Issues/PRs

Example: Fixes #18911.

What does this implement/fix?

Add function pinball_error as a new regression scoring function used to estimate quantile regression with quantile != 0.5.

@sdpython sdpython changed the title Add pinball_error metrics, fixes issue #18911 [WIP] Add pinball_error metrics, fixes issue #18911 Feb 9, 2021
glemaitre
glemaitre reviewed Feb 9, 2021
View changes
sklearn/metrics/__init__.py Outdated Show resolved Hide resolved
@sdpython sdpython changed the title [WIP] Add pinball_error metrics, fixes issue #18911 Add pinball_error metrics, fixes issue #18911 Feb 9, 2021
ogrisel
ogrisel reviewed Feb 9, 2021
View changes
Copy link
Member

@ogrisel ogrisel left a comment

Thanks @sdpython! Here is a first pass of review comments:

doc/modules/classes.rst Outdated Show resolved Hide resolved
doc/modules/model_evaluation.rst Outdated Show resolved Hide resolved
doc/modules/model_evaluation.rst Outdated Show resolved Hide resolved
doc/modules/model_evaluation.rst Show resolved Hide resolved
sklearn/metrics/_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/_regression.py Show resolved Hide resolved
sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
ogrisel
ogrisel reviewed Feb 10, 2021
View changes
doc/modules/model_evaluation.rst Outdated Show resolved Hide resolved
ogrisel
ogrisel reviewed Feb 10, 2021
View changes
Copy link
Member

@ogrisel ogrisel left a comment

Thanks, I find the updated example very interesting.

sklearn/ensemble/tests/test_gradient_boosting_loss_functions.py Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
@ogrisel ogrisel changed the title Add pinball_error metrics, fixes issue #18911 Add pinball_loss metric for quantile regression, fixes issue #18911 Feb 11, 2021
ogrisel
ogrisel reviewed Feb 15, 2021
View changes
doc/modules/model_evaluation.rst Outdated Show resolved Hide resolved
@ogrisel
Copy link
Member

@ogrisel ogrisel commented Feb 15, 2021

@sdpython @lorentzenchr I pushed the renaming in my last commits. I think I also addressed most of the pending review comments. Let me know what you think.

@ogrisel ogrisel changed the title Add pinball_loss metric for quantile regression, fixes issue #18911 Add mean_pinball_loss metric for quantile regression, fixes issue #18911 Feb 15, 2021
lorentzenchr
lorentzenchr reviewed Feb 15, 2021
View changes
Copy link
Member

@lorentzenchr lorentzenchr left a comment

@ogrisel Accompanying you improving examples is a pleasure and a lot of fun. Now I learned how to highlight cells in displayed pandas tables😃

sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Show resolved Hide resolved
@ogrisel
Copy link
Member

@ogrisel ogrisel commented Feb 15, 2021

Now I learned how to highlight cells in displayed pandas tables😃

I googled after your comment and found this solution on stackoverflow :)

@ogrisel ogrisel requested a review from lorentzenchr Feb 16, 2021
lorentzenchr
lorentzenchr approved these changes Feb 16, 2021
View changes
Copy link
Member

@lorentzenchr lorentzenchr left a comment

LGTM

sklearn/metrics/tests/test_regression.py Show resolved Hide resolved
sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
@lorentzenchr
Copy link
Member

@lorentzenchr lorentzenchr commented Feb 18, 2021

The CI failure of test_imports_strategies in macOS pylatest_conda_forge_mkl seems unrelated. Therefore, I dare to merge...

@lorentzenchr lorentzenchr changed the title Add mean_pinball_loss metric for quantile regression, fixes issue #18911 ENH Add mean_pinball_loss metric for quantile regression Feb 18, 2021
@lorentzenchr lorentzenchr merged commit 6a6217f into scikit-learn:main Feb 18, 2021
24 of 26 checks passed
@lorentzenchr lorentzenchr mentioned this pull request Feb 18, 2021
Open
4 tasks
@ogrisel
Copy link
Member

@ogrisel ogrisel commented Feb 19, 2021 •
edited

Thanks for the PR @sdpython! and @lorentzenchr for the reviews :)

@glemaitre glemaitre mentioned this pull request Apr 22, 2021
Merged
12 tasks
@ghost
Copy link

@ghost ghost commented May 26, 2021

Should this be median pinball loss for alpha=0.5/tau=0.5 (as per https://www.tensorflow.org/addons/api_docs/python/tfa/losses/pinball_loss)?

@lorentzenchr
Copy link
Member

@lorentzenchr lorentzenchr commented May 26, 2021

This PR implemented the mean of the pinball loss as a metric, which equals 1/2 * absolute loss for alpha=0.5 (which elicits the median). I think the tensorflow fomula has a typo (i.e. is incorrect).

@airmilesabdullah
Copy link

@airmilesabdullah airmilesabdullah commented Dec 22, 2021

hi there, I noticed that when my pandas dataframe has more than 10,000 rows, I get an out-of-memory error. Why does this occur and is there a way around it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ogrisel ogrisel

@glemaitre glemaitre

@lorentzenchr lorentzenchr

Assignees
No one assigned
Labels
module:ensemble module:metrics Waiting for Reviewer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add quantile loss as metric
6 participants
@sdpython @ogrisel @lorentzenchr @GaelVaroquaux @airmilesabdullah @glemaitre