Across the whole site? I’d say no.
For visual rhythym, I’d aim for consistent sizing on the same page area etc. For instance if you have a row of three buttons, I’d expect them to be the same size.
People like symmetry so jarring differences in button sizes are generally detracting from the message you are trying to convey.
You might have button further down that page by itself not related to others and that could be a different size if required.
