As this is painful and partly impossible to build with Homeassistant automations, I've created a small service that directly talks to matter-server's websocket and makes my BILRESA dual button + scroll wheel remotes work the way I expect.
Disclaimer: This is a proof of concept vibecoded in a few nights and it works for me™️.
https://codeberg.org/markuslindenberg/home-remote-controller