The Wayback Machine - https://web.archive.org/web/20210911024254/https://github.com/arduino/arduino-cli/pull/1333
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Pluggable Discoveries #1333

Merged
merged 74 commits into from Aug 23, 2021
Merged

Add support for Pluggable Discoveries #1333

merged 74 commits into from Aug 23, 2021

Conversation

@silvanocerza
Copy link
Contributor

@silvanocerza silvanocerza commented Jun 23, 2021

Please check if the PR fulfills these requirements

  • The PR has no duplicates (please search among the Pull Requests
    before creating one)
  • The PR follows
    our contributing guidelines
  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • UPGRADING.md has been updated with a migration guide (for breaking changes)
  • What kind of change does this PR introduce?

This PR introduces quite a bit of changes to add support for Pluggable Discoveries as specified by the related RFC.

  • What is the current behavior?

Currently the Arduino CLI uses a bundled serial-discovery tool to search for supported boards connected to serial ports, there is no support for boards connected to the network or 3rd party boards that require specific tools to be discovered.

There is currently no way for 3rd party platforms to specify a custom tool to run to discover boards.

  • What is the new behavior?

After this PR is merged any platform will be able to specify custom tools to run at discovery time to find supported boards, for example when calling arduino-cli board list.

This change is fully retro compatible with platform that don't specify a discovery tool, the bundled serial-discovery tool will be used as a fallback.

Discovery tools will be internally loaded automatically when the Arduino CLI is initialized and run when necessary.

Yes, they are documented in the docs/UPGRADING.md file.

  • Other information:

None.


See how to contribute

@silvanocerza silvanocerza requested a review from arduino/team_tooling Jun 23, 2021
@silvanocerza silvanocerza self-assigned this Jun 23, 2021
@cmaglie cmaglie force-pushed the scerza/pluggable-tools branch 3 times, most recently from e381a03 to 6f2b9b8 Jul 2, 2021
cmaglie added a commit that referenced this pull request Jul 6, 2021
… in #1333 (#1351)

* Fix discovery client

* Fix client_example

* Fixed BoardWatch message loop

* Upgraded go-properties-orderedmap to v1.5.0
@cmaglie cmaglie force-pushed the scerza/pluggable-tools branch 3 times, most recently from a8c5220 to 6e0b14d Jul 6, 2021
@silvanocerza silvanocerza force-pushed the scerza/pluggable-tools branch 7 times, most recently from c5a2184 to a4594a8 Jul 8, 2021
@silvanocerza silvanocerza changed the title Add loading of third party pluggable discoveries Add support for Pluggable Discoveries Jul 21, 2021
@cmaglie cmaglie force-pushed the scerza/pluggable-tools branch from 1ab82ed to e6de840 Jul 21, 2021
@silvanocerza silvanocerza requested a review from per1234 Jul 22, 2021
@silvanocerza silvanocerza marked this pull request as ready for review Jul 22, 2021
@silvanocerza silvanocerza force-pushed the scerza/pluggable-tools branch from 8d389c4 to 5ab1ecd Jul 22, 2021
@per1234
Copy link
Contributor

@per1234 per1234 commented Jul 22, 2021

I get a panic now if I try to upload without specifying a port:

$ ./arduino-cli version
arduino-cli.exe alpha Version: git-snapshot Commit: 5ab1ecde Date: 2021-07-22T16:58:19Z

$ ./arduino-cli compile -u -b arduino:avr:uno /tmp/FooSketch
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0xd4c6e9]

goroutine 1 [running]:
github.com/arduino/arduino-cli/cli/compile.run(0xc000755600, 0xc00011d380, 0x1, 0x4)
        C:/Users/per/go/src/github.com/arduino/arduino-cli/cli/compile/compile.go:206 +0x969
github.com/spf13/cobra.(*Command).execute(0xc000755600, 0xc00011d340, 0x4, 0x4, 0xc000755600, 0xc00011d340)
        C:/Users/per/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc00014fb80, 0x0, 0xc000210c60, 0x0)
        C:/Users/per/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        C:/Users/per/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
        C:/Users/per/go/src/github.com/arduino/arduino-cli/main.go:31 +0x8f

Previously there was a friendly error message:

$ ./arduino-cli version
arduino-cli.exe alpha Version: git-snapshot Commit: fc367b77 Date: 2021-07-22T16:55:44Z

$ ./arduino-cli compile -u -b arduino:avr:uno /tmp/FooSketch
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
Error during Upload: uploading error: no upload port provided

arduino/cores/packagemanager/loader.go Outdated Show resolved Hide resolved
cli/arguments/reference.go Outdated Show resolved Hide resolved
cli/arguments/reference.go Outdated Show resolved Hide resolved
docs/UPGRADING.md Outdated Show resolved Hide resolved
docs/platform-specification.md Outdated Show resolved Hide resolved
docs/platform-specification.md Outdated Show resolved Hide resolved
docs/platform-specification.md Outdated Show resolved Hide resolved
docs/platform-specification.md Outdated Show resolved Hide resolved
docs/platform-specification.md Outdated Show resolved Hide resolved
@silvanocerza
Copy link
Contributor Author

@silvanocerza silvanocerza commented Jul 23, 2021

I get a panic now if I try to upload without specifying a port:

$ ./arduino-cli version
arduino-cli.exe alpha Version: git-snapshot Commit: 5ab1ecde Date: 2021-07-22T16:58:19Z

$ ./arduino-cli compile -u -b arduino:avr:uno /tmp/FooSketch
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0xd4c6e9]

goroutine 1 [running]:
github.com/arduino/arduino-cli/cli/compile.run(0xc000755600, 0xc00011d380, 0x1, 0x4)
        C:/Users/per/go/src/github.com/arduino/arduino-cli/cli/compile/compile.go:206 +0x969
github.com/spf13/cobra.(*Command).execute(0xc000755600, 0xc00011d340, 0x4, 0x4, 0xc000755600, 0xc00011d340)
        C:/Users/per/go/pkg/mod/github.com/spf13/[email protected]/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc00014fb80, 0x0, 0xc000210c60, 0x0)
        C:/Users/per/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        C:/Users/per/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
        C:/Users/per/go/src/github.com/arduino/arduino-cli/main.go:31 +0x8f

Previously there was a friendly error message:

$ ./arduino-cli version
arduino-cli.exe alpha Version: git-snapshot Commit: fc367b77 Date: 2021-07-22T16:55:44Z

$ ./arduino-cli compile -u -b arduino:avr:uno /tmp/FooSketch
Sketch uses 444 bytes (1%) of program storage space. Maximum is 32256 bytes.
Global variables use 9 bytes (0%) of dynamic memory, leaving 2039 bytes for local variables. Maximum is 2048 bytes.
Error during Upload: uploading error: no upload port provided

That's no good. It should now be possible to upload without specifying a port for boards that use tools that automatically detected the correct port, but it certainly must not crash like this for those that don't.

Thanks for the great review anyway. 🙏

@silvanocerza silvanocerza force-pushed the scerza/pluggable-tools branch from e6edb87 to ca6ca3d Jul 23, 2021
docs/pluggable-discovery-specification.md Outdated Show resolved Hide resolved
docs/pluggable-discovery-specification.md Outdated Show resolved Hide resolved
docs/pluggable-discovery-specification.md Outdated Show resolved Hide resolved
docs/pluggable-discovery-specification.md Outdated Show resolved Hide resolved
docs/pluggable-discovery-specification.md Outdated Show resolved Hide resolved
docs/pluggable-discovery-specification.md Outdated Show resolved Hide resolved
docs/pluggable-discovery-specification.md Outdated Show resolved Hide resolved
mkdocs.yml Outdated Show resolved Hide resolved
rpc/cc/arduino/cli/commands/v1/upload.proto Outdated Show resolved Hide resolved
rpc/cc/arduino/cli/commands/v1/upload.proto Outdated Show resolved Hide resolved
@rsora rsora mentioned this pull request Jul 26, 2021
docs/platform-specification.md Outdated Show resolved Hide resolved
docs/platform-specification.md Outdated Show resolved Hide resolved
docs/pluggable-discovery-specification.md Outdated Show resolved Hide resolved
@silvanocerza silvanocerza force-pushed the scerza/pluggable-tools branch from ccdc20d to bf84915 Jul 28, 2021
This was linked to issues Aug 20, 2021
test/test_upload_mock.py Outdated Show resolved Hide resolved
test/test_upload_mock.py Outdated Show resolved Hide resolved
test/test_upload_mock.py Outdated Show resolved Hide resolved
@cmaglie cmaglie force-pushed the scerza/pluggable-tools branch 2 times, most recently from 6544e1b to c5f14e0 Aug 23, 2021
@cmaglie cmaglie force-pushed the scerza/pluggable-tools branch 4 times, most recently from 6f63cf5 to 8ef520c Aug 23, 2021
@cmaglie cmaglie force-pushed the scerza/pluggable-tools branch from 8ef520c to 40943b1 Aug 23, 2021
@cmaglie cmaglie merged commit ec027a7 into master Aug 23, 2021
100 checks passed
100 checks passed
@github-actions
run-determination
Details
@github-actions
run-determination
Details
@github-actions
run-determination
Details
@github-actions
run-determination
Details
@github-actions
lint
Details
@github-actions
lint
Details
@github-actions
check
Details
@github-actions
run-determination
Details
@github-actions
lint
Details
@github-actions
check
Details
@github-actions
run-determination
Details
@github-actions
run-determination
Details
@github-actions
run-determination
Details
@github-actions
run-determination
Details
@github-actions
run-determination
Details
@github-actions
run-determination
Details
@github-actions
check-errors (./)
Details
@github-actions
check-errors (./)
Details
@github-actions
check
Details
@github-actions
check
Details
@github-actions
links
Details
@github-actions
links
Details
@github-actions
build
Details
@github-actions
formatting
Details
@github-actions
build
Details
@github-actions
build
Details
@github-actions
test (ubuntu-latest)
Details
@github-actions
test (ubuntu-latest)
Details
@github-actions
test (ubuntu-latest)
Details
@github-actions
test (ubuntu-latest)
Details
@github-actions
check-errors (arduino/discovery/discovery_client)
Details
@github-actions
check-errors (arduino/discovery/discovery_client)
Details
@github-actions
test (windows-latest)
Details
@github-actions
test (windows-latest)
Details
@github-actions
test (windows-latest)
Details
@github-actions
test (windows-latest)
Details
@github-actions
check-errors (client_example)
Details
@github-actions
check-errors (client_example)
Details
@github-actions
test (macos-latest)
Details
@github-actions
test (macos-latest)
Details
@github-actions
test (macos-latest)
Details
@github-actions
test (macos-latest)
Details
@github-actions
check-errors (commands/daemon/term_example)
Details
@github-actions
check-errors (commands/daemon/term_example)
Details
@github-actions
check-errors (docsgen)
Details
@github-actions
check-errors (docsgen)
Details
@github-actions
check-outdated (./)
Details
@github-actions
check-outdated (./)
Details
@github-actions
check
Details
@github-actions
checksums artifact
Details
@github-actions
checksums artifact
Details
@github-actions
check-outdated (arduino/discovery/discovery_client)
Details
@github-actions
check-outdated (arduino/discovery/discovery_client)
Details
@github-actions
Linux_X86-32 artifact
Details
@github-actions
Linux_X86-32 artifact
Details
@github-actions
check-outdated (client_example)
Details
@github-actions
check-outdated (client_example)
Details
@github-actions
Linux_X86-64 artifact
Details
@github-actions
Linux_X86-64 artifact
Details
@github-actions
check-outdated (commands/daemon/term_example)
Details
@github-actions
check-outdated (commands/daemon/term_example)
Details
@github-actions
Linux_ARM64 artifact
Details
@github-actions
Linux_ARM64 artifact
Details
@github-actions
check-outdated (docsgen)
Details
@github-actions
check-outdated (docsgen)
Details
@github-actions
Linux_ARMv6 artifact
Details
@github-actions
Linux_ARMv6 artifact
Details
@github-actions
Linux_ARMv7 artifact
Details
@github-actions
Linux_ARMv7 artifact
Details
@github-actions
macOS_64 artifact
Details
@github-actions
macOS_64 artifact
Details
@github-actions
Windows_X86-32 artifact
Details
@github-actions
Windows_X86-32 artifact
Details
@github-actions
Windows_X86-64 artifact
Details
@github-actions
Windows_X86-64 artifact
Details
@github-actions
check-style (./)
Details
@github-actions
check-style (./)
Details
@github-actions
check-formatting
Details
@github-actions
clean
Details
@github-actions
clean
Details
@github-actions
check-style (arduino/discovery/discovery_client)
Details
@github-actions
check-style (arduino/discovery/discovery_client)
Details
@github-actions
check-style (client_example)
Details
@github-actions
check-style (client_example)
Details
@github-actions
check-style (commands/daemon/term_example)
Details
@github-actions
check-style (commands/daemon/term_example)
Details
@github-actions
check-style (docsgen)
Details
@github-actions
check-style (docsgen)
Details
@github-actions
check-formatting (./)
Details
@github-actions
check-formatting (./)
Details
@github-actions
check-formatting (arduino/discovery/discovery_client)
Details
@github-actions
check-formatting (arduino/discovery/discovery_client)
Details
@github-actions
check-formatting (client_example)
Details
@github-actions
check-formatting (client_example)
Details
@github-actions
check-formatting (commands/daemon/term_example)
Details
@github-actions
check-formatting (commands/daemon/term_example)
Details
@github-actions
check-formatting (docsgen)
Details
@github-actions
check-formatting (docsgen)
Details
@github-actions
check-config (./)
Details
@github-actions
check-config (./)
Details
@cmaglie cmaglie deleted the scerza/pluggable-tools branch Aug 23, 2021
@silvanocerza
Copy link
Contributor Author

@silvanocerza silvanocerza commented Aug 23, 2021

🎉

@rsora
Copy link
Contributor

@rsora rsora commented Aug 23, 2021

image

@matthijskooijman
Copy link
Collaborator

@matthijskooijman matthijskooijman commented Sep 10, 2021

I noticed a regression that I expect is caused by this PR. When using arduino-cli from git now, uploads using (our slightly customized version of) the STM32 core fail with:

Error during Upload: Property 'upload.tool.serial' is undefined

The board in question indeed does not define upload.tool.serial, but still uses the pre-pluggable syntax of defining upload.tool, which should be translated to upload.tool.default by convertUploadToolsToPluggableDiscovery(). However, this core does not define upload.tool on the board directly, but defines it in a board menu: https://github.com/stm32duino/Arduino_Core_STM32/blob/974b35690cbf4680ce5746c61c69fb20935bae2f/boards.txt#L522

To reproduce:

$ arduino-cli compile --fqbn STMicroelectronics:stm32:Nucleo_32:pnum=NUCLEO_F031K6,upload_method=serialMethod --upload --port /dev/ttyACM0 
Sketch uses 7612 bytes (23%) of program storage space. Maximum is 32768 bytes.
Global variables use 828 bytes (20%) of dynamic memory, leaving 3268 bytes for local variables. Maximum is 4096 bytes.
Error during Upload: Property 'upload.tool.serial' is undefined

I guess this is fairly easy to fix in the core, though (just define both upload.tool.default (or upload.tool.<protocol>) in addition to the legacy upload.tool, but it might be useful to fix compatibility here too. An obvious fix would be to implement the fallback when actually doing the upload, rather than when loading the boards).

@fpistm, you might find this interesting too :-)

@cmaglie
Copy link
Member

@cmaglie cmaglie commented Sep 10, 2021

@matthijskooijman you're my nightmare :-)

An obvious fix would be to implement the fallback when actually doing the upload, rather than when loading the boards).

yes I think it's the thing to do, I'll move your comment in a new issue, so we can keep track of it.

@fpistm
Copy link

@fpistm fpistm commented Sep 10, 2021

@matthijskooijman you're my nightmare :-)

@cmaglie Mine too 😜

@matthijskooijman
Copy link
Collaborator

@matthijskooijman matthijskooijman commented Sep 10, 2021

Lol, thanks for the compliments :-p

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

6 participants