Use JIT Plug-in for coverage to cover JIT'd functions and methods #56310

janeyx99 · 2021-04-16T22:59:22Z

This PR is step 2 (after #56708) to having JIT coverage--it actually uses the plug-in in CI!

Disclaimer: note that this will mark the entire JIT'd function/method as covered without seeking proof that the
compiled code has been executed. This means that even if the code chunk is merely compiled and not run, it will get
marked as covered.

Test plan:
We should see coverage improvements in CI after. A file to look out for would be torch/jit/quantized.py, which should have more coverage after this PR, which it does!
https://codecov.io/gh/pytorch/pytorch/src/d3283ccd8c898e7a1c5b46179a0b9147c87c53b9/torch/jit/quantized.py vs https://codecov.io/gh/pytorch/pytorch/src/master/torch/jit/quantized.py

More generally, the whole jit folder got ~3% increase in coverage, I believe.

The text was updated successfully, but these errors were encountered:

facebook-github-bot · 2021-04-16T22:59:29Z

💊 CI failures summary and remediations

As of commit d3283cc (more details on the Dr. CI page):

2/3 failures possibly* introduced in this PR
- 1/2 non-scanned failure(s)
1/3 broken upstream at merge base 3a4344a on Apr 22 from 2:25pm to 4:40pm

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_bionic_py3_8_gcc9_coverage_test1 (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Apr 23 18:35:07 Build left local git repository checkout dirty

Apr 23 18:35:06 + assert_git_not_dirty
Apr 23 18:35:06 + [[ pytorch-linux-bionic-py3.8-gcc9-coverage-test1 != *rocm* ]]
Apr 23 18:35:06 + [[ pytorch-linux-bionic-py3.8-gcc9-coverage-test1 != *xla* ]]
Apr 23 18:35:06 ++ git status --porcelain
Apr 23 18:35:07 + git_status='?? test/my_test/'
Apr 23 18:35:07 + [[ -n ?? test/my_test/ ]]
Apr 23 18:35:07 + echo 'Build left local git repository checkout dirty'
Apr 23 18:35:07 + echo 'git status --porcelain:'
Apr 23 18:35:07 + echo '?? test/my_test/'
Apr 23 18:35:07 + exit 1
Apr 23 18:35:07 Build left local git repository checkout dirty
Apr 23 18:35:07 git status --porcelain:
Apr 23 18:35:07 ?? test/my_test/
Apr 23 18:35:07 + cleanup
Apr 23 18:35:07 + retcode=1
Apr 23 18:35:07 + set +x
Apr 23 18:35:07 =================== sccache compilation log ===================
Apr 23 18:35:07 =========== If your build fails, please take a look at the log above for possible reasons ===========
Apr 23 18:35:07 Compile requests                     12
Apr 23 18:35:07 Compile requests executed             6
Apr 23 18:35:07 Cache hits                            6

🚧 1 fixed upstream failure:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

If your commit is older than viable/strict, run these commands:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

binary_linux_libtorch_3_7m_cpu_gcc5_4_cxx11-abi_shared-with-deps_build on Apr 22 from 2:25pm to 4:40pm (3a44d26 - febff45)
- 🔁 rerun

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-bionic-rocm4.1-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

codecov · 2021-04-21T07:09:53Z

Codecov Report

Merging #56310 (5164858) into master (a462634) will decrease coverage by 0.00%.
The diff coverage is n/a.

❗ Current head 5164858 differs from pull request most recent head cb4690d. Consider uploading reports for the commit cb4690d to get more accurate results

@@            Coverage Diff             @@
##           master   #56310      +/-   ##
==========================================
- Coverage   77.77%   77.77%   -0.01%     
==========================================
  Files        1924     1924              
  Lines      190977   190977              
==========================================
- Hits       148534   148528       -6     
- Misses      42443    42449       +6

Summary: This PR is step 1 to covering JIT'd methods and functions. Step 2 (using it in CI) is here: #56310. 1. This PR introduces a package `coverage_plugins` that hosts JITPlugin. 2. We also bring in a `.coveragerc` file that is used in CI to omit the files we don't want to report on (e.g., temporary directories or test or utils.) **Disclaimer: This PR does NOT use the plug-in. Nothing should change as a result.** Pull Request resolved: #56708 Test Plan: CI. Coverage should not go down. If you're interested in testing this plug-in locally, you should: `pip install -e tools/coverage_plugins_package` from the root directory. Add the following lines to `.coveragerc` under `[run]` ``` plugins = coverage_plugins.jit_plugin ``` And then try: `coverage run test/test_jit.py TestAsync.test_async_script_no_script_mod` You should see `.coverage.jit` show up at the end. You can then run `coverage combine --append` and `coverage debug data` to see that some files in `torch/jit` are covered. Reviewed By: samestep Differential Revision: D27945570 Pulled By: janeyx99 fbshipit-source-id: 78732940fcb498d5ec37d4075c4e7e08e96a8d55

janeyx99 · 2021-04-22T22:21:12Z

The conflicts are with myself--I will rebase, but wanted to see if CI passed first

walterddr · 2021-04-22T22:26:22Z

it should improve in general in torch/jit/* right? not just quantized.py

janeyx99 · 2021-04-22T22:27:56Z

it should improve in general in torch/jit/* right? not just quantized.py

Yes actually I'm not even seeing the changes at all...hmmmm let me debug this before making you review haha.

I have discovered why! it's cuz we use coverage-4 instead of 5, and COVERAGE_RCFILE is coverage 5 only.

samestep · 2021-04-23T14:54:47Z

.circleci/docker/common/install_conda.sh

@@ -102,6 +102,7 @@ if [ -n "$ANACONDA_PYTHON_VERSION" ]; then
  # TODO: Why is scipy pinned
  # Pin MyPy version because new errors are likely to appear with each release
  # Pin hypothesis to avoid flakiness: https://github.com/pytorch/pytorch/issues/31136
+  # Pin coverage so we can use COVERAGE_RCFILE


Does this mean that we expect COVERAGE_RCFILE to be removed in future versions of coverage? Or is the purpose of this just to modify the script in some way so that the Docker image gets rebuilt?

Ah, no, this was because our previously installed version of coverage did not have the COVERAGE_RCFILE option, so this explains why we pinned it in the first place

facebook-github-bot · 2021-04-23T15:39:55Z

@janeyx99 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2021-04-23T20:19:40Z

@janeyx99 merged this pull request in 88bd051.

facebook-github-bot · 2021-04-24T14:55:04Z

This pull request has been reverted by 3fbc154.

facebook-github-bot · 2021-04-26T15:46:07Z

@janeyx99 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…torch#56310) Summary: This PR is step 2 (after pytorch#56708) to having JIT coverage--it actually uses the plug-in in CI! Disclaimer: note that this will mark the entire JIT'd function/method as covered without seeking proof that the compiled code has been executed. This means that even if the code chunk is merely compiled and not run, it will get marked as covered. Pull Request resolved: pytorch#56310 Test Plan: We should see coverage improvements in CI after. A file to look out for would be `torch/jit/quantized.py`, which should have more coverage after this PR, which it does! https://codecov.io/gh/pytorch/pytorch/src/d3283ccd8c898e7a1c5b46179a0b9147c87c53b9/torch/jit/quantized.py vs https://codecov.io/gh/pytorch/pytorch/src/master/torch/jit/quantized.py More generally, the whole jit folder got ~3% increase in coverage, I believe. Reviewed By: walterddr Differential Revision: D28000672 Pulled By: janeyx99 fbshipit-source-id: 6712979d63a5e1224a92ee9bd9679ec62cf1cbba

Summary: This PR is step 1 to covering JIT'd methods and functions. Step 2 (using it in CI) is here: pytorch#56310. 1. This PR introduces a package `coverage_plugins` that hosts JITPlugin. 2. We also bring in a `.coveragerc` file that is used in CI to omit the files we don't want to report on (e.g., temporary directories or test or utils.) **Disclaimer: This PR does NOT use the plug-in. Nothing should change as a result.** Pull Request resolved: pytorch#56708 Test Plan: CI. Coverage should not go down. If you're interested in testing this plug-in locally, you should: `pip install -e tools/coverage_plugins_package` from the root directory. Add the following lines to `.coveragerc` under `[run]` ``` plugins = coverage_plugins.jit_plugin ``` And then try: `coverage run test/test_jit.py TestAsync.test_async_script_no_script_mod` You should see `.coverage.jit` show up at the end. You can then run `coverage combine --append` and `coverage debug data` to see that some files in `torch/jit` are covered. Reviewed By: samestep Differential Revision: D27945570 Pulled By: janeyx99 fbshipit-source-id: 78732940fcb498d5ec37d4075c4e7e08e96a8d55

…torch#56310) Summary: This PR is step 2 (after pytorch#56708) to having JIT coverage--it actually uses the plug-in in CI! Disclaimer: note that this will mark the entire JIT'd function/method as covered without seeking proof that the compiled code has been executed. This means that even if the code chunk is merely compiled and not run, it will get marked as covered. Pull Request resolved: pytorch#56310 Test Plan: We should see coverage improvements in CI after. A file to look out for would be `torch/jit/quantized.py`, which should have more coverage after this PR, which it does! https://codecov.io/gh/pytorch/pytorch/src/d3283ccd8c898e7a1c5b46179a0b9147c87c53b9/torch/jit/quantized.py vs https://codecov.io/gh/pytorch/pytorch/src/master/torch/jit/quantized.py More generally, the whole jit folder got ~3% increase in coverage, I believe. Reviewed By: ezyang Differential Revision: D27967517 Pulled By: janeyx99 fbshipit-source-id: 53fd8431d772c2447191135c29d1b166ecd42f50

…torch#56310) Summary: This PR is step 2 (after pytorch#56708) to having JIT coverage--it actually uses the plug-in in CI! Disclaimer: note that this will mark the entire JIT'd function/method as covered without seeking proof that the compiled code has been executed. This means that even if the code chunk is merely compiled and not run, it will get marked as covered. Pull Request resolved: pytorch#56310 Test Plan: We should see coverage improvements in CI after. A file to look out for would be `torch/jit/quantized.py`, which should have more coverage after this PR, which it does! https://codecov.io/gh/pytorch/pytorch/src/d3283ccd8c898e7a1c5b46179a0b9147c87c53b9/torch/jit/quantized.py vs https://codecov.io/gh/pytorch/pytorch/src/master/torch/jit/quantized.py More generally, the whole jit folder got ~3% increase in coverage, I believe. Reviewed By: walterddr Differential Revision: D28000672 Pulled By: janeyx99 fbshipit-source-id: 6712979d63a5e1224a92ee9bd9679ec62cf1cbba

facebook-github-bot added the cla signed label Apr 16, 2021

janeyx99 force-pushed the play-with-coverage branch 4 times, most recently from d59bd2c to 5164858 Apr 21, 2021

janeyx99 force-pushed the play-with-coverage branch 3 times, most recently from bf20e9f to deb8702 Apr 22, 2021

janeyx99 mentioned this pull request Apr 22, 2021

Introducing JitPlugin #56708

Closed

janeyx99 force-pushed the play-with-coverage branch from deb8702 to 62066fb Apr 22, 2021

janeyx99 changed the title ~~[skip-ci] First JIT coverage plug-in commit~~ Use JIT Plug-in for coverage to cover JIT'd functions and methods Apr 22, 2021

janeyx99 marked this pull request as ready for review Apr 22, 2021

janeyx99 requested a review from Apr 22, 2021

janeyx99 marked this pull request as draft Apr 22, 2021

janeyx99 added 2 commits Apr 22, 2021

Use JitPlugin in CI

71bd1e8

upgrade coverage to 5.5 to use COVERAGE_RCFILE

d3283cc

janeyx99 force-pushed the play-with-coverage branch from 62066fb to d3283cc Apr 22, 2021

janeyx99 marked this pull request as ready for review Apr 23, 2021

janeyx99 requested a review from gmagogsfm Apr 23, 2021

walterddr approved these changes Apr 23, 2021

View changes

samestep approved these changes Apr 23, 2021

View changes

facebook-github-bot closed this in 88bd051 Apr 23, 2021

facebook-github-bot added the Merged label Apr 23, 2021

facebook-github-bot added the Reverted label Apr 24, 2021

janeyx99 mentioned this pull request Apr 26, 2021

Name .coverage.jit with timestamp to prevent loss of stats #56829

Closed

janeyx99 reopened this Apr 26, 2021

facebook-github-bot closed this in a90a3ac Apr 26, 2021

janeyx99 mentioned this pull request Jul 29, 2021

[jit] TorchScript does not work with Python coverage package #43146

Closed

Oct	NOV	Feb
	08
2020	2021	2022

pytorch / pytorch Public

Use JIT Plug-in for coverage to cover JIT'd functions and methods #56310

Use JIT Plug-in for coverage to cover JIT'd functions and methods #56310

janeyx99 commented Apr 16, 2021 •

edited

Loading

facebook-github-bot commented Apr 16, 2021 •

edited

Loading

codecov bot commented Apr 21, 2021 •

edited

Loading

janeyx99 commented Apr 22, 2021

walterddr commented Apr 22, 2021

janeyx99 commented Apr 22, 2021 •

edited

Loading

samestep Apr 23, 2021

janeyx99 Apr 23, 2021

facebook-github-bot commented Apr 23, 2021

facebook-github-bot commented Apr 23, 2021

facebook-github-bot commented Apr 24, 2021

facebook-github-bot commented Apr 26, 2021

pytorch / pytorch Public

Use JIT Plug-in for coverage to cover JIT'd functions and methods #56310

Use JIT Plug-in for coverage to cover JIT'd functions and methods #56310

Conversation

janeyx99 commented Apr 16, 2021 • edited Loading

facebook-github-bot commented Apr 16, 2021 • edited Loading

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

pytorch_linux_bionic_py3_8_gcc9_coverage_test1 (1/1)

🚧 1 fixed upstream failure:

ci.pytorch.org: 1 failed

codecov bot commented Apr 21, 2021 • edited Loading

Codecov Report

janeyx99 commented Apr 22, 2021

walterddr commented Apr 22, 2021

janeyx99 commented Apr 22, 2021 • edited Loading

samestep Apr 23, 2021

janeyx99 Apr 23, 2021

facebook-github-bot commented Apr 23, 2021

facebook-github-bot commented Apr 23, 2021

facebook-github-bot commented Apr 24, 2021

facebook-github-bot commented Apr 26, 2021

janeyx99 commented Apr 16, 2021 •

edited

Loading

facebook-github-bot commented Apr 16, 2021 •

edited

Loading

codecov bot commented Apr 21, 2021 •

edited

Loading

janeyx99 commented Apr 22, 2021 •

edited

Loading