Description
Use case
The upcoming Dart MCP server wants to be able to expose screenshotting capability to LLMs through an MCP tool. There are many potential use cases:
- Allow the LLM to validate its changes visually (if you ask to change the color of a button, after hot reloading did the color actually change as expected?).
- Prompt the LLM to improve the visual style of your app, this allows it to see what it currently looks like in order to suggest changes.
- Allows you to ask the LLM to edit something by describing it visually, it can screenshot the current page to understand which thing you are talking about.
Proposal
We should have a VM service method that can reliably take screenshots on any device, probably the _flutter.screenshot extension method. This would allow us to take screenshots for a specific app on any device.
Current state
Calling the _flutter.screenshot VM service method (the non-skia specific extension), returns _flutter.screenshot: (-32000) Could not capture image screenshot.
Similarly, running flutter screenshot --type device --vm-service-url <url> currently crashes with:
Oops; flutter has exited unexpectedly: "_flutter.screenshotSkp: (-32000) Cannot capture SKP screenshot with Impeller enabled.".
This is using a different extension method that seems skia specific, if we do get impeller support we might want to change this tool to invoke the more general version.
Ideally, one of these would be implemented on impeller (the former one doesn't say specifically that it is an impeller issue but I think it likely is?).
Why not device screenshots?
The upcoming Dart MCP server links up to your apps by VM service URI, and doesn't know the device type, so --type device isn't a great option (we would have to guess the device type or list all devices and require the user to select the device type).
Today, we only get screenshot support through the vm service method by default on desktop devices though, so that means the AI can't validate its changes visually any other device.
Other options
Get device name over VM service
We could potentially do a device screenshot if we could get the device name over the vm service, so possibly flutter tool could register an extension method to get the device id? This might be a simpler solution if it is challenging to implement screenshots on impeller.
Take device screenshots but identified by VM service URI
Alternatively, the flutter screenshot command could be able to take device screenshots, but for a specific app identified by its VM service URI, instead of a device ID.