There's a couple ways to go about this:
First check your recognition history by saying "show recognition history" to see which specific word (or number) Dragon 'hears' when you are not saying anything.
Is it "4" (like with the click by voice Chrome extension) or is it "four" (written form)?
If I'm not mistaken you can change the spoken form of the number 4 into something else like do-four, i-four etc. I'm not sure if spaces/dashes are allowed, Ron can tell you more about that.
Did you check your audio settings? You could let Dragon check it or you could use a free program like Audacity to record the microphone you are currently using. Press record and don't say anything. The sound input (visually) should be a very thin line, without any glitches. If the recording contains any audible humming or other sounds then Dragon may get confused. Cable may be damaged, sound input may be too loud etc.
However, the solution for these kinds of problems is sometimes as easy as simply creating a new Dragon user profile.