I know this is an old question but since it doesn't have an accepted answer, here is a comparison between the two tools. Searching online you almost always find recommendations to install both. Someone can feel free to tell me I'm wrong here but I think that's a waste of resources.
The current version of ModSecurity 2.9.3 with CRS 3.3.1 can in my opinion take the place of running mod_evasive. I think it does a better job and is more configurable although the two are slightly different.
mod_evasive by default will block requests from an IP address for the same Apache site when it hits 50/second from that IP. It also has a separate setting to block requests from an IP addr for the "same page" on a site. This defaults to 2/second which is going to be way to low for most sites. The default blocking period is 10s (low in my opinion). Of course all of that can be configured. The main difference between mod_evasive and mod_security is that mod_evasive also blocks "same page" requests by default. I should also mention that mod_evasive uses the Apache child process number to determine if the requests are counted against the IP addr. If Apache spawns a new process then those requests will not count towards a block. Also, some requests could slip through even after a block. You will want to make sure Apache has a high MaxRequestsPerChild value.
mod_security with Core Rule Set installed does not protect against DOS attacks by default. You have to specifically enable that functionality in the crs-setup.conf file by uncommenting rule 900700:
SecAction \
"id:900700,\
phase:1,\
nolog,\
pass,\
t:none,\
setvar:'tx.dos_burst_time_slice=60',\
setvar:'tx.dos_counter_threshold=100',\
setvar:'tx.dos_block_timeout=600'"
Here is what it does: by default this will block requests from a specific IP address when a client makes 100 or more requests within 60s. So for instance, if a client makes 100 requests in say 30s they will be blocked for 600s. There are other settings that affect this: If setvar:tx.paranoia_level=1 then two bursts of 100 requests, each within 60s are required before blocking occurs. However, if you set 'setvar:tx.paranoia_level=2' or greater then only one burst is required.
ModSecurity also adds another nice piece of default functionality: it excludes static files from the request counter. In rule 900260 the default extensions are: setvar:'tx.static_extensions=/.jpg/ /.jpeg/ /.png/ /.gif/ /.js/ /.css/ /.ico/ /.svg/ /.webp/' This helps prevent accidental blocking when a client is requesting a static file since these do not require huge resources from your server. That way we focus on DOS attacks that could actually slow down the server while reducing false positives.
Conclusion
If want to easily control blocking by specific page requests or you need quick, easy to set up DOS protection then enable mod_evasive. I'll also add that it's probably a bit easier to configure mod_evasive to update your iptables and run scripts on a hit.
If you want more control over every aspect of how DOS attacks are blocked and what happens during a block then you really only need mod_security. If all you care about is DOS attacks then mod_security is probably overkill. In terms of a DDoS attack, I think both tools are going to be equally useful with default settings since they are each storing a hash table by IP addr. Now, if you want to write specific rules for mod_security then there is no limit to how and what you block and you could even recreate the block by page request functionality of mod_evasive if you think you need it.
