OK, lets define download. Downloading means transferring something to the browser. The browser then either displays the content (like images) if it can, or asks to save it to disk (if it cannot display it).
So int this case, if you would have "preview", the image would already have to be downloaded to the browser.
If it is not the preview you are wondering about, but rather would want to get rid of the "X"-image, then it seems like tough luck. The isAllowed-method does not seem to be available in the template, and thus you cannot use it to say something like "if I have the right to see that there's an attachmtn available, show a link, but if I also have the download-permission, also show a thumbnail".
I tested this, and could not use the isAllowed-method in the template succesfully. So changing this is out of my scope at least.