temper, tamper

[DmitryL]

Hello,
My question is to the natives. Could you actually distinguish a difference between 'temper' and 'tamper' pronounciation? I saw the transcription and I know there is some, but can't catch it while listening.
Would it be very annoying if in my speech there were only one sound for 'tamper' and 'temper', 'man' and 'men', 'kettle' and 'cattle'?
May be you could provide me some tips of how should one say it? Should I try to make 'e' sound in 'tamper' to be something between 'u' as in 'sudden' and 'e' as in 'rest'?
Thanks,

Dmitry