18X Speedup in Wordfence Scan

Wordfence 6.2.0 was released yesterday and it includes something really special: a huge improvement in scan performance. I’m going to share with you what changed and why Wordfence 6.2.0 is the same great quality malware and security scan running up to 18X faster.

Malware scanning is a critical part of the layered approach to protection that Wordfence provides. The Wordfence Firewall stops attacks in their tracks. Scanning is how we determine if a hacker has gained access to your site. You’ll recall that we’ve also integrated the malware scan into the Wordfence Firewall to scan file uploads as they arrive. This gives you three layers of protection against attacks.

Our Firewall has always been extremely fast and the malware scan built into the firewall uses the same model of execution. While we are always looking for ways to optimize our firewall code, we really don’t need to improve in that area because it’s so darn fast already.

However, we had received reports from a few users that scanning would take quite a long time on larger sites. We have also had reports from a few hosting providers that when a large number of scans are running concurrently (200 or more), they could use a lot of disk IO resources.

So we decided to tackle this issue head-on and work closely with our customers and hosting providers.

Working to make scan faster

I met with one large host in person and had a very informative and collaborative discussion (Thanks Heidi!!) that included follow-up conversations with our team and a direct channel of communication with our devs and QA team. Then we put our heads down and got to work.

Ryan Britton, one of our senior developers at Wordfence led the technical effort and Matt Rusnak led the QA effort on this project. They went away and worked on this and when they came back with the numbers, I couldn’t believe the results.

scanperformance3

To measure performance, we isolated the process running Wordfence and monitored disk reads. The above bar charts show the amount of data read in megabytes for the older version of Wordfence compared to 6.2.0 which is in green. Less is better because it reduces the amount of IO load on the physical server hosting the site running Wordfence.

  • For Wordfence 6.2.0 we halved the IO load on sites running a brand new plain WordPress installation.
  • On a small site of around 92MB we achieved a 3.6X reduction in disk IO load.
  • On a larger site with 682MB we are seeing an impressive 18.4X reduction in disk IO load.

Larger sites will see an even greater increase in speed and reduction in IO load during scans. We don’t know what the average size of a WordPress site is, but we estimate that it’s similar to the large site model we used. My personal blog, for example, is 552MB of files and it’s been around for a reasonable amount of time.

Scan functionality and detection capability stays exactly the same.

We haven’t reduced the number of signatures we’re using for detection or the number of files we read. There was a bugfix included in this release where we no longer scan .txt files unless you have ‘High Sensitivity’ scanning enabled, but that is unrelated to performance improvements.

How we made these huge performance gains

We achieved this huge speed increase by changing the way we’re performing MD5 and SHA256 hashes of files, getting smarter about not hashing files where we aren’t using the hash and reducing the hashing operation to a single streaming read on each file.

We also removed counters that were not needed that were increasing file reads. In addition, this new release keeps better track of which directories have been scanned so that we don’t need to reiterate through a large list of directories when the scan resumes.

It’s worth noting that image-heavy sites will see a particularly large improvement in scan speed.

…And then we went even further!

One of the amazing things about the Wordfence team is that they are super passionate about what they do. For 6.2.0 I was expecting to get a modest increase in scan speed. Instead we got a huge gain in scan speed. Then the customer service team got involved and encouraged Dev and QA to go even further.

What they came up with is what we’re calling a “Low resource scan”. When this option is enabled, Wordfence does exactly the same amount of work, but it introduces an artificial delay in the scan which consumes no additional resources but spreads the disk IO load out over time. This reduces peak usage load on servers that have slow disk or that are hosting a lot of other busy websites.

You can enable the “Low resource scan” by going to the Wordfence options page on your WordPress site, scroll down to the “Scans to Include” heading and then at the bottom of that section you can find the option labeled “Use low resource scanning. Reduces server load by lengthening the scan duration.”. You can find the documentation for that option here.

As you can tell from above, we have massively reduced the amount of disk IO that Wordfence generates, but just in case that’s not enough and your hosting provider asks you to reduce things further, you can now enable this option.

In our tests, the low resource scan takes 1.7 to 3.7 times longer to run, depending on your site, while reducing peak load. Again, it does the same amount of work, just spread over more time.

The low resource scan combined with our much faster scan engine gives you a very low impact way to regularly scan for malware while keeping your hosting provider happy and your site super fast. We recommend most people leave this setting off, but in extreme cases of limited resources, customers can now enable this setting.

We recommend the following configuration:

You will find the best performance gains when Wordfence is using the default scan configuration. For our own sites we leave low-resource scanning off and just use the default Wordfence options. The scan is now incredibly fast.

On my own blog with 6127 files totaling 552MB of space, my scan now finishes in 24 seconds from a cold start to finish. I run my personal site on the cheapest $10/month Linode server available. That is incredibly fast on a very modest server.

We think the standard configuration will work great for the vast majority of sites out there.

In Conclusion

As always a huge thanks from me to our incredible team for this collaborative effort. And a very special thanks to the customers and hosting providers who worked with us to help us better understand their needs.

Wordfence remains committed to securing the WordPress community and platform. To that end we work closely with our customers and hosting providers to ensure that we continue to provide the best protection available for WordPress. This is another major leap forward in achieving our goal of a secure WordPress community and a more secure Web.

As always we welcome your feedback in the comments below, or you can contact us using one of the channels on our contact page.

Did you enjoy this post? Share it!

Comments

14 Comments
  • Thanks alot .

  • Wow - 18x is some optimisation work. Wish my developers were that good!

  • We noticed a massive uptick in unexplainable mysql activity for a period of time yesterday.. The only thing connected to mysql is our wordpress blog, however we didn't get a lot of visits. Could the wordfence plugin somehow have caused this? It's not a problem, I just don't like unexplained sustained network spikes due to mysql communication.

    • No it probably wasn't us.

  • This is a HUGE improvement and likely removes my biggest reservation about deploying Wordfence on all my clients' sites (typically hosted on GoDaddy shared Linux hosting) -- that the I/O Usage would spike when a Wordfence scan ran and GoDaddy would send out repeated emails to the client encouraging them to upgrade to a higher level of resources (when I know they didn't needed it). And now I shouldn't have to exclude "*.jpg" and "*.png" on graphics-intensive sites. Excellent!

    Keep up the good work! You guys are AMAZING!!!

  • Hi Mark. As a hosting provider, a BIG THANKS for this! The WordFence scans were VERY noticeable and I'm looking forward to seeing users upgrade to 6.2. :-)

  • well done. Good work.

  • Just completed a scan and it was very noticeably faster! A massive improvement in performance so well done.

    Just a quick question though... the end of the scan was "Scanning for weak passwords". I thought all passwords were passed through a one-way algorithm before being put in the database so how can you tell from a hashed password whether it's weak or not?

    Thanks for all your hard work on this plugin. You and your team have done brilliantly.

    • Hi John,

      I haven't looked at this code and I did ping the team. But AFAIK we take a list of extremely common passwords which I think is in wfDict.php in our source and hash them using the salt the same way WP does. Then compare the result to what's in the DB. If the hashes match then we include that account in the results with a warning that they have an extremely common password. That's how cracking of hashed passwords is done in general.

      To learn more about password hashing, salts, etc we have an extremely detailed article I wrote in our learning center: https://www.wordfence.com/learn/how-passwords-work-and-cracking-passwords/

      Mark.

  • Now that's customer service. Taking a great product but not being satisfied and improving it even further.

    Thanks and congratulations to all the team - great effort!

    Jonathan

  • Earlier this year I tried using Wordfence and on 2 of my 3 sites it kept giving me fatal errors on the scan. I worked with my web host (LiquidWeb, great support), and I have a dedicated server, to make sure I had enough memory, but even 256MB would not handle it. So I had to unfortunately had to stop using it. Will this speed improvement also improve memory usage and perhaps not give the the fatal error?

    • Yes it should also reduce memory footprint. Give it a try and please use our support forums if you need further assistance. Thanks.

  • I manage about 70 websites, all using Wordfence. The new speed is great, but there are a few sites that still take hours for a scan to complete. Yes, low resource scanning is NOT checked. All my sites use the same theme (Divi) and generally the same plugins. Any idea why some sites have problems and others don't? Deleting the tables on deactivation and then activating does not help. If anyone can help, I'd appreciate it, as I'm sure some other folks will as well.