The anti-virus software scans all the existing files in your directory and cross-checks with known malware. If they match, they are blocked.
If its a modified version of an already existing virus, the behavioural pattern or content of these files would match on some level with already know virus.
If the malware is something that has not been discovered or is a new virus, then, the anti-virus checks for patterns in the file that may indicate if its a virus or not, but more often than not, these viruses are caught when you are downloading the file itself.