The Wayback Machine - https://web.archive.org/web/20210221144500/https://github.com/scikit-learn/scikit-learn/pull/19415
Skip to content

/ scikit-learn

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH Add mean_pinball_loss metric for quantile regression #19415

Merged
merged 63 commits into from Feb 18, 2021
Merged

ENH Add mean_pinball_loss metric for quantile regression #19415

merged 63 commits into from Feb 18, 2021
+595 −65

Conversation

@sdpython
Copy link
Contributor

@sdpython sdpython commented Feb 9, 2021

Reference Issues/PRs

Example: Fixes #18911.

What does this implement/fix?

Add function pinball_error as a new regression scoring function used to estimate quantile regression with quantile != 0.5.
@sdpython
add pinball_error
Loading status checks…
468c156
@sdpython sdpython changed the title Add pinball_error metrics, fixes issue #18911 [WIP] Add pinball_error metrics, fixes issue #18911 Feb 9, 2021
@glemaitre
glemaitre reviewed Feb 9, 2021
View changes
sklearn/metrics/__init__.py Outdated Show resolved Hide resolved
sdpython added 2 commits Feb 9, 2021
@sdpython
fix \r issue
05dca44
@sdpython
add \r
Loading status checks…
367b265
sdpython added 4 commits Feb 9, 2021
@sdpython
add \r
Loading status checks…
d710a6f
@sdpython
remove \r
Loading status checks…
72d2d51
@sdpython
fix lint issue
Loading status checks…
5bc2bc5
@sdpython
lint
Loading status checks…
621abdd
@sdpython sdpython changed the title [WIP] Add pinball_error metrics, fixes issue #18911 Add pinball_error metrics, fixes issue #18911 Feb 9, 2021
@sdpython
lint
Loading status checks…
67f3c02
@ogrisel
ogrisel reviewed Feb 9, 2021
View changes
Copy link
Member

@ogrisel ogrisel left a comment

Thanks @sdpython! Here is a first pass of review comments:
doc/modules/classes.rst Outdated Show resolved Hide resolved
doc/modules/model_evaluation.rst Outdated Show resolved Hide resolved
doc/modules/model_evaluation.rst Outdated Show resolved Hide resolved
doc/modules/model_evaluation.rst Show resolved Hide resolved
sklearn/metrics/_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/_regression.py Show resolved Hide resolved
sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
sdpython added 6 commits Feb 9, 2021
@sdpython
rename pinball_error into pinball_loss
Loading status checks…
789442c
@sdpython
check exception is raised
Loading status checks…
076452d
@sdpython
Fix failing unit test
Loading status checks…
359538e
@sdpython
refactor example on gradient boosting
Loading status checks…
1c2d6d2
@sdpython
lint
Loading status checks…
ef64dd0
@sdpython
lint
Loading status checks…
9e56a19
@ogrisel
ogrisel reviewed Feb 10, 2021
View changes
doc/modules/model_evaluation.rst Outdated Show resolved Hide resolved
sdpython added 2 commits Feb 10, 2021
@sdpython
add dependency on tqdm for examples, fix documentation
Loading status checks…
f07ad17
@sdpython
add new unit test
Loading status checks…
c41f2e3
@ogrisel
ogrisel reviewed Feb 10, 2021
View changes
Copy link
Member

@ogrisel ogrisel left a comment

Thanks, I find the updated example very interesting.
sklearn/ensemble/tests/test_gradient_boosting_loss_functions.py Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
@ogrisel
ogrisel reviewed Feb 10, 2021
View changes
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
sdpython added 4 commits Feb 10, 2021
@sdpython
improve example, add another test on pinball_error with sample_weights
Loading status checks…
1bdc94e
@sdpython
lint
Loading status checks…
0aa34df
@sdpython
fix failing test due to very small discrepencies
Loading status checks…
31d721c
@sdpython
documentation
Loading status checks…
f8988c7
@ogrisel ogrisel changed the title Add pinball_error metrics, fixes issue #18911 Add pinball_loss metric for quantile regression, fixes issue #18911 Feb 11, 2021
ogrisel and others added 5 commits Feb 15, 2021
@ogrisel
Assess coverage
Loading status checks…
070cbab
@ogrisel @lorentzenchr
Apply suggestions from code review
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
ecf1a95 
Co-authored-by: Christian Lorentzen <[email protected]>
@ogrisel
Stricter tests for regression metrics
4ce9196
@ogrisel
Minimize pinball loss with Nelder-Mead
Loading status checks…
1b2d0ed
@ogrisel
Small fixes and improvements in model_evaluation.rst
Loading status checks…
f6482e6
@ogrisel
ogrisel reviewed Feb 15, 2021
View changes
doc/modules/model_evaluation.rst Outdated Show resolved Hide resolved
ogrisel added 3 commits Feb 15, 2021
@ogrisel
Rename variable in test
Loading status checks…
bc1882a
@ogrisel
Rename pinball_loss to mean_pinball_loss
Loading status checks…
53e0230
@ogrisel
Fix linter
Loading status checks…
6694b1f
@ogrisel
Copy link
Member

@ogrisel ogrisel commented Feb 15, 2021

@sdpython @lorentzenchr I pushed the renaming in my last commits. I think I also addressed most of the pending review comments. Let me know what you think.
ogrisel added 3 commits Feb 15, 2021
@ogrisel
Fix missing indent in math formula
Loading status checks…
d139dc2
@ogrisel
Fix phrasing
Loading status checks…
728d632
@ogrisel
Add integration test
Loading status checks…
eb22059
@ogrisel ogrisel changed the title Add pinball_loss metric for quantile regression, fixes issue #18911 Add mean_pinball_loss metric for quantile regression, fixes issue #18911 Feb 15, 2021
@lorentzenchr
lorentzenchr reviewed Feb 15, 2021
View changes
Copy link
Contributor

@lorentzenchr lorentzenchr left a comment

@ogrisel Accompanying you improving examples is a pleasure and a lot of fun. Now I learned how to highlight cells in displayed pandas tables😃
sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Outdated Show resolved Hide resolved
examples/ensemble/plot_gradient_boosting_quantile.py Show resolved Hide resolved
@ogrisel
Copy link
Member

@ogrisel ogrisel commented Feb 15, 2021

Now I learned how to highlight cells in displayed pandas tables😃

I googled after your comment and found this solution on stackoverflow :)
ogrisel added 4 commits Feb 15, 2021
@ogrisel
Change optimization test to make it run faster
8edbed1
@ogrisel
Missing cell marker
1b36537
@ogrisel
Missing comas
Loading status checks…
85ab9d3
@ogrisel
DOC small improvements
Loading status checks…
70d9323
@ogrisel ogrisel requested a review from lorentzenchr Feb 16, 2021
@lorentzenchr
lorentzenchr approved these changes Feb 16, 2021
View changes
Copy link
Contributor

@lorentzenchr lorentzenchr left a comment

LGTM
sklearn/metrics/tests/test_regression.py Show resolved Hide resolved
sklearn/metrics/tests/test_regression.py Outdated Show resolved Hide resolved
@ogrisel @lorentzenchr
Update sklearn/metrics/tests/test_regression.py
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
7820618 
Co-authored-by: Christian Lorentzen <[email protected]>
@lorentzenchr
Copy link
Contributor

@lorentzenchr lorentzenchr commented Feb 18, 2021

The CI failure of test_imports_strategies in macOS pylatest_conda_forge_mkl seems unrelated. Therefore, I dare to merge...
@lorentzenchr lorentzenchr changed the title Add mean_pinball_loss metric for quantile regression, fixes issue #18911 ENH Add mean_pinball_loss metric for quantile regression Feb 18, 2021
@lorentzenchr lorentzenchr merged commit 6a6217f into scikit-learn:main Feb 18, 2021
24 of 26 checks passed
24 of 26 checks passed
triage
Details
labeler
Details
Check build trigger
Details
Build wheel for cp${{ matrix.python }}-${{ matrix.platform_id }}-${{ matrix.manylinux_image }}
Details
Source distribution
Details
Upload to Anaconda
Details
scikit-learn.scikit-learn Build #20210218.2 had test failures
Details
scikit-learn.scikit-learn (macOS pylatest_conda_forge_mkl) macOS pylatest_conda_forge_mkl failed
Details
LGTM analysis: C/C++ No code changes detected
Details
LGTM analysis: JavaScript No code changes detected
Details
LGTM analysis: Python 1 new alert
Details
ci/circleci: deploy Your tests passed on CircleCI!
Details
ci/circleci: doc Your tests passed on CircleCI!
Details
ci/circleci: doc artifact Link to 0/doc/_changed.html
Details
ci/circleci: doc-min-dependencies Your tests passed on CircleCI!
Details
ci/circleci: lint Your tests passed on CircleCI!
Details
scikit-learn.scikit-learn (Get Git Commit) Get Git Commit succeeded
Details
scikit-learn.scikit-learn (Linting) Linting succeeded
Details
scikit-learn.scikit-learn (Linux py36_conda_openblas) Linux py36_conda_openblas succeeded
Details
scikit-learn.scikit-learn (Linux py36_ubuntu_atlas) Linux py36_ubuntu_atlas succeeded
Details
scikit-learn.scikit-learn (Linux pylatest_pip_openblas_pandas) Linux pylatest_pip_openblas_pandas succeeded
Details
scikit-learn.scikit-learn (Linux32 py36_ubuntu_atlas_32bit) Linux32 py36_ubuntu_atlas_32bit succeeded
Details
scikit-learn.scikit-learn (Linux_Runs pylatest_conda_mkl) Linux_Runs pylatest_conda_mkl succeeded
Details
scikit-learn.scikit-learn (Windows py36_pip_openblas_32bit) Windows py36_pip_openblas_32bit succeeded
Details
scikit-learn.scikit-learn (Windows py37_conda_mkl) Windows py37_conda_mkl succeeded
Details
scikit-learn.scikit-learn (macOS pylatest_conda_mkl_no_openmp) macOS pylatest_conda_mkl_no_openmp succeeded
Details
@lorentzenchr lorentzenchr mentioned this pull request Feb 18, 2021
Open
1 of 4 tasks complete
@ogrisel
Copy link
Member

@ogrisel ogrisel commented Feb 19, 2021 •
edited

Thanks for the PR @sdpython! and @lorentzenchr for the reviews :)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ogrisel ogrisel

@glemaitre glemaitre

@lorentzenchr lorentzenchr

Assignees
No one assigned
Labels
Waiting for Reviewer module:ensemble module:metrics
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

Add quantile loss as metric
5 participants
@sdpython @ogrisel @lorentzenchr @GaelVaroquaux @glemaitre