I rebooted my Ooma Telo 3 (2018) and the voicemail indicator light and banner now showed on all 3 handsets.
(To reboot your Ooma got to: http://172.27.35.1/system.lp and click "Reboot")
I listened to and deleted the voicemail on handset #3 and the voicemail light and banner turned off on all 3 handsets.
This has happened many times and a reboot always fixes it.
On other occasions after I have listened to and deleted the voicemail I will see the voicemail indicator on 1 or 2 of the 3 handsets but there is no voicemail. Again, a reboot of the Ooma always fixes it.
My Ooma is attached directly to my Internet modem.