Mark Twain said it took him 3 weeks to write an impromptu speech. Churchill was accused of spending much of his free time writing unprepared speeches. But when you are known as a good public speaker, people expect you speak without warning or preparation. To decline to say an impromptu few words would be surprising and may even harm your reputation.
My experience with the impromptu has always been when the expected speaker had failed to turn up or had to leave early. I have made presentations to people I didn’t know, honoured efforts I had no knowledge of and thanked contributors I could not identify. With keyword notes hastily assembled from the well-informed (who felt unable to speak in public themselves) I would say the required few words, I hope with enthusiasm.
It’s worth assembling your own small collection of “unprepared” welcome/thanks/congratulation texts as they will prove useful and are a further way to increase your skills.