The Wayback Machine - https://web.archive.org/web/20210104141300/https://github.com/bazelbuild/rules_python/pull/373
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce compile_pip_requirements rule #373

Open
wants to merge 1 commit into
base: master
from

Conversation

@alexeagle
Copy link
Collaborator

@alexeagle alexeagle commented Oct 19, 2020

This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes #176

Copy link
Collaborator

@thundergolfer thundergolfer left a comment

Thanks for upstreaming.

To give others some added context, we at Canva also use pip-tools to 'compile' our top-level deps into a locked transitive closure, but do this with a less elegant bash script called regenerate_py_deps.sh. The script updates a .txt that then becomes the input file to pip_install.

Over the year we've used pip-tools for this purpose, we've found it practical and reliable, though it is annoying to need a third-party tool to do something so essential.

examples/pip_install/BUILD Outdated Show resolved Hide resolved
python/pip_install/pip_requirements.bzl Outdated Show resolved Hide resolved
pip==9.0.3
setuptools==44.0.0
wheel==0.30.0a0
#

This comment has been minimized.

@thundergolfer

thundergolfer Oct 25, 2020
Collaborator

We'd be able to get rid of this requirements file once piptool is fully removed from the rules. Would we then remove the macro usage here, or is there a reason to keep it?

This comment has been minimized.

@alexeagle

alexeagle Oct 26, 2020
Author Collaborator

sure if the requirements file goes away, then so should the rule that keeps it updated.

python/pip_install/pip_requirements.bzl Outdated Show resolved Hide resolved
@alexeagle alexeagle force-pushed the alexeagle:pip_requirements branch 4 times, most recently from d01ae74 to 74f80f0 Oct 26, 2020
@alexeagle
Copy link
Collaborator Author

@alexeagle alexeagle commented Oct 26, 2020

ah, now the CI failure is because it's running python 2 and one of the requirements varies with python version

Collecting funcsigs>=1; python_version < "3.3" (from mock==2.0.0->-r /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/rules_python/python/requirements.txt (line 7))
--
  | (In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
  | funcsigs>=1; python_version < "3.3" from
@alexeagle alexeagle force-pushed the alexeagle:pip_requirements branch 2 times, most recently from 9f192e0 to 6bfa3c0 Oct 26, 2020
@alexeagle alexeagle changed the title Introduce pip_requirements rule Introduce compile_pip_requirements rule Nov 2, 2020
@alexeagle alexeagle force-pushed the alexeagle:pip_requirements branch 2 times, most recently from a800010 to d6984b1 Nov 2, 2020
This uses pip-tools to compile a requirements.in file to a requirements.txt file,
allowing transitive dependency versions to be pinned so that builds are reproducible.

Fixes #176
@alexeagle alexeagle force-pushed the alexeagle:pip_requirements branch from d6984b1 to 7175da7 Nov 2, 2020
@dhalperi
Copy link
Contributor

@dhalperi dhalperi commented Nov 6, 2020

Hey, I tried to be sneaky and try this out. Did I do it wrong?

  1. Update rules_python to this commit, use it in //python (where my requirements.txt is).
diff --git a/WORKSPACE b/WORKSPACE
index 06724b178..15306f4af 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -23,10 +23,10 @@ register_toolchains("//python/interpreter:bfe_py_toolchain")
 ### Load rules_python first so our version wins.
 http_archive(
     name = "rules_python",
-    sha256 = "95ee649313caeb410b438b230f632222fb5d2053e801fe4ae0572eb1d71e95b8",
-    strip_prefix = "rules_python-c8c79aae9aa1b61d199ad03d5fe06338febd0774",
+    sha256 = "72800a67f1aa25d13dfeb8269e1d92c2dfb044792893605238db90b6a0c39a3c",
+    strip_prefix = "rules_python-7175da78d6d032094ea69ab418bc7f8e6f15573a",
     # equivalent SHA to that of 0.1.0 release, except the archive has experimental stuff like wheel.bzl
-    url = "https://github.com/bazelbuild/rules_python/archive/c8c79aae9aa1b61d199ad03d5fe06338febd0774.tar.gz",
+    url = "https://github.com/bazelbuild/rules_python/archive/7175da78d6d032094ea69ab418bc7f8e6f15573a.tar.gz",
 )
 
 load("@rules_python//python:repositories.bzl", "py_repositories")
diff --git a/python/BUILD b/python/BUILD
index ffd0fb0cd..020ee0cc5 100644
--- a/python/BUILD
+++ b/python/BUILD
@@ -1 +1,3 @@
 package(default_visibility = ["//visibility:public"])
+load("@rules_python//python/pip_install:requirements.bzl", "compile_pip_requirements")
+compile_pip_requirements()
  1. Then I went into //python and copied requirements.txt to requirements.in.

  2. bazel run :requirements.update in //python.

I got this error:

INFO: Invocation ID: 400b5a31-5f0b-487b-8bc5-275ae4a61b1f
DEBUG: /private/var/tmp/_bazel_dhalperin/95eb8687a0f56fe58f756c786043f2ca/external/rules_python/python/repositories.bzl:8:10: py_repositories is a no-op and is deprecated. You can remove this from your WORKSPACE file
DEBUG: /private/var/tmp/_bazel_dhalperin/95eb8687a0f56fe58f756c786043f2ca/external/rules_python/python/legacy_pip_import/pip.bzl:143:10: DEPRECATED: the pip_import rule has been replaced with pip_install, please see rules_python 0.1 release notes
INFO: Analyzed target //python:requirements.update (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //python:requirements.update up-to-date:
  bazel-bin/python/requirements.update
INFO: Elapsed time: 0.307s, Critical Path: 0.02s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/python/requirements.update python/requirements.in python/requirements.txt requireINFO: Build completed successfully, 1 total action
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_dhalperin/95eb8687a0f56fe58f756c786043f2ca/execroot/batfish_enterprise/bazel-out/darwin-fastbuild/bin/python/requirements.update.runfiles/rules_python/python/pip_install/pip_compile.py", line 7, in <module>
    from piptools.scripts.compile import cli
  File "/private/var/tmp/_bazel_dhalperin/95eb8687a0f56fe58f756c786043f2ca/execroot/batfish_enterprise/bazel-out/darwin-fastbuild/bin/python/requirements.update.runfiles/pypi__pip_tools/piptools/scripts/compile.py", line 17, in <module>
    from .._compat import parse_requirements
  File "/private/var/tmp/_bazel_dhalperin/95eb8687a0f56fe58f756c786043f2ca/execroot/batfish_enterprise/bazel-out/darwin-fastbuild/bin/python/requirements.update.runfiles/pypi__pip_tools/piptools/_compat/__init__.py", line 5, in <module>
    import six
ModuleNotFoundError: No module named 'six'

FWIW, we are using a custom python interpreter (3.7.9) and our python runtime has no python2 allowed.

Also, sorry for jumping in randomly... don't want to distract too badly from the point of the PR, but if this is a bug (missing dep?) it might be helpful.

@thundergolfer
Copy link
Collaborator

@thundergolfer thundergolfer commented Nov 12, 2020

I've now test-driven this myself and run into the same error as @dhalperi. As yet I haven't investigated the issue deeply, but I can create the same import problem in my workspace's Nix shell with PYTHONPATH="" python3. six is not stdlib, so I think this is just a dependency of pip_tools that is undeclared in the PR.


Unrelated, but I it was a bit finicky figuring out whether to create the requirements.txt file when setting this up (as an empty file), or let the tool do it. Eg.

bazel build //tools/build/python/...         
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured).
INFO: Found 2 targets...
ERROR: /Users/jonathon/Code/thundergolfer/foo/tools/build/python/BUILD:3:25: //tools/build/python:requirements.update: missing input file '//tools/build/python:requirements.txt'
ERROR: /Users/jonathon/Code/thundergolfer/foo/tools/build/python/BUILD:3:25 1 input file(s) do not exist
INFO: Elapsed time: 0.115s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
FAILED: Build did NOT complete successfully

So at some point bootstrap instructions would be good to tell users how to smoothly onboard into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants
You can’t perform that action at this time.