How we dress for the occasion of receiving the body, blood, soul and divinity of Jesus Christ does matter. Do you dress for the occasion?
How you dress reflects your attitude and respect you have toward the person you have chosen to be with, in this case God.
If these down dressers were invited to attend the presidential inauguration of President Barack Hussein Obama, would they dress as if they were going to work in the yard?
So the question boils down to: Do you really believe Christ is present? One can only logically conclude you do not if you do not dress for the occasion and present yourself so as to witness the world you came to receive Christ. If you do not believe he is truly present, what difference does it make how you are dressed?