It's funny that you asked this question exactly when I left for a conference this past week, otherwise I would have gotten on this answer sooner!
In broad terms, AV does a number of things:
- Scans the files on disk (hard drive, SSD, USB storage etc...) and looks for "suspicious things." These suspicious things can be actual known signatures of malware which were picked up and previously submitted to the community by a malware researcher or a malware research tool OR they can be known methods of suspicious behavior. For example, rules can be made such that certain OS functions which, when combined in certain ways, often are used for malicious reasons, and therefore the AV software may flag the file as being bad. This is why you often get false positives with AV software. This general method is good when AV software is installed for the first time, to locate malware that may already exist on the computer before the AV was installed.
- Scans each file just as it's going to be loaded in memory by the operating system file loader - What happens if you download malware and you haven't done a full scan of your system with anti-malware software before you try to start the newly downloaded program? Without this feature, the computer would be compromised by the malware. What AV does in this case is every single file you launch gets scanned real quick and if anything suspicious is found (similar to above*), the loading process will be terminated and the AV will warn you, and may even remove the file and put it into the quarantine. This means that if you try to launch known malware, the AV will stop it from launching and protect your computer. Something called "hooking" makes this possible. Without getting into too much detail, it's possible to freeze a process just as it begins to execute a certain function. Since the OS always follows the same routine to start the file loading process, AVs will "hook" this function and then examine the file at this stage before it is loaded into memory and executed.
- Observes network traffic - Some AVs do this more than others, however, the AV system may also monitor the network traffic and if it finds traffic associated with known bad exploits or malware, it will take action. For example, say a user downloads malware which is sending out a ton of spam email behind the user's back secretly. The AV may detect this and flag the program as malware.
- Scans Files already loaded in memory - AV can also scan files which are loaded in memory periodically, looking for suspicious code libraries and executables which are running in memory. Known-good processes may also be scanned because many times, malware injects itself into legitimate processes on the computer to hide.
- Examine web browser activity - More and more AVs have web browsing plugins because the web is often the source of malware to begin with. These plugins may stop the user from entering known bad sites which distribute malware, scan downloaded files before they are allowed to even be opened by the user, and/or watch for malicious scripting. Scripting attacks are bad because sometimes they can compromise a system without the user even downloading anything at all! If you want to see how, read this article.
- Using a whitelist - This is probably the safest approach but it can also generate more false positives. This means that _any file which the AV has not explicitly marked as "good" is automatically blocked by default. _The sweet thing about this is that no definitions are necessary because the file is either on the whitelist or it's not. The challenge is that when a new file is created and is not yet on the whitelist, it will be blocked unless there is special logic making an exception for the file. Whitelists are a more conservative approach and frankly, are the most effective for the self-mutating malware out there which is constantly changing hashes.
*SPECIAL NOTE: You may wonder how on earth the AV could scan a huge file and check it for virus signatures so quickly. One way this is done is using hashes. I can't explain hashes entirely here (look them up if you don't know), but basically, each malware is run through a hashing algorithm like SHA-256 for example. This algorithm creates a special unique number (the hash) for the bytes that make up the file. It is very important to note that the hash has absolutely nothing to do with the file name but instead, the entire contents of the file are quickly fed through the algorithm to create the unique hash. What this means is that known bad malware will always have the same hash unless the file bytes change. So often, a quick way for AV to determine if a file is bad is to hash it and then check that hash number with a database. If the hash is in the bad file database, it will be flagged as bad without even having to go through all of the above detailed checks which could require more time for the CPU to complete.
It is important to note that if an exe's hash matches a bad file, the file is 99.99% chance the bad file. However, just because a file's hash doesn't match any database entries, doesn't mean the file is clean.. It just means that this particular byte arrangement in this file is different from previously submitted ones. This means that if a virus can change itself and spread, it will create a unique hash every single time and then won't be picked up by AV which only checks hashes.