Updates on CyberSecurity, WordPress and what we're cooking in the lab today.

Wordfence Launches Short-Circuit Scan Signatures – Up to 6X Performance Increase

This entry was posted in Wordfence on August 30, 2017 by Mark Maunder   11 Replies

In October 2016, the Wordfence team started chatting about a way to radically boost the speed of scans once we grow beyond a certain number of scan signatures. As a reminder, a scan signature is a pattern that recognizes a certain kind of malware.

Today Wordfence has 4,523 signatures available for the free community, and we have an additional 226 new signatures that are only available to Wordfence Premium users. These become free once they are 30 days old.

New Malware Constantly Emerging

Our team continuously adds from 30 to over 100 new scan signatures each week. The site cleaning team constantly discovers new kinds of malware as they clean hacked websites, and each malware sample is turned into a scan signature and released to Wordfence to help it detect that malware.

Wordfence is currently at a total of 4,749 scan signatures for our Premium customers (4,523 free + 226 Premium), and within one year, this will grow to somewhere between 6,000 to 10,000 signatures at the current rate we are discovering new malware.

The constant increase in the amount of malware targeting WordPress is clear, and Wordfence needs to continually grow our scan signatures to keep pace.

Radical Innovation to Address Growth in WordPress Malware

In October last year, Matt Rusnak, who heads up QA for Wordfence, and Ryan Britton, who is our senior core developer for Wordfence, started chatting about a new algorithm to radically speed up the Wordfence scan and make it able to handle a much larger number of signatures.

Matt suggested identifying common patterns across scan signatures, grouping those signatures together and then checking if a file contains the common pattern first before scanning with the signatures in each group.

In theory, if we had 10,000 signatures, and if we are able to identify groups of 100 scan signatures and create a pre-check for each one, we would need to match only 100 scan signatures for every item we scanned instead of 10,000. Ryan dubbed this “short-circuiting.”

Earlier this year, we started work on the project. We created the services to support short-circuiting, then tested and launched them on our back-end servers about a month ago. Support was released for short-circuiting within the Wordfence plugin in the past few weeks. And we have been working to create grouped scan signatures with common “short-circuit” patterns.

Matt Barry, our lead developer who created the Wordfence firewall, worked with Ryan and Matt Rusnak to make this project happen, and they received help from other members of the engineering team.

A 2X to 6X Speedup With Short-Circuit Scanning

When we released Wordfence 6.3.17 late last week, you may have noticed an entry in the changelog which said, “Improvement: Prepared code for upcoming scan improvement which will greatly increase scan performance by optimizing malware signatures.

On Monday this week, we enabled short-circuit scanning on our servers. The speed improvement in Wordfence scans was breathtaking, to say the least.

We are seeing a 2-to-6-times performance increase across our test sites and customer sites. This is an incredible improvement.

On one major hosting provider, scans on one of our large test sites went from an average scan time of 8 minutes per scan to 1 minute and 20 seconds for a Wordfence scan to complete.

Continuous Engineering Innovation in WordPress Security

This is not the first time we have radically improved scan speed. Last year in September we increased scan speed by refactoring the way we perform many operations in the scan.

In July of this year, we further improved scan performance for hosting providers by monitoring scan distribution across hosting provider VPS instances and introducing a smoothing algorithm.

Short-circuiting scan signatures is a powerful new technique the team has created to provide a radical performance improvement on an already fast scan.

While our many of our competitors don’t even provide a firewall and malware scan in their security products, the Wordfence engineering team is at the forefront of engineering innovation, ensuring that you benefit from a powerful firewall and malware scan combination with lighting-fast performance.

Congratulations and thank you to Matt Rusnak, Ryan Britton, Matt Barry and Åsa Rosenberg, who all contributed to bringing short-circuit scanning to our customers.

Did you enjoy this post? Share it!


4.50 (20 votes) Your rating:

11 Comments on "Wordfence Launches Short-Circuit Scan Signatures – Up to 6X Performance Increase"

Reine Johansson August 30, 2017 at 9:14 am • Reply

Congratulations! There's such a pleasure in finding new solutions to acheive massive performance gains!

Ismail Ozdemir August 30, 2017 at 9:20 am • Reply

You guys are rocking it again. Well done to team!

Bronson Harrington August 30, 2017 at 9:50 am • Reply

Congratulations on a monumental achievement. This is a massive performance boost thanks to some clever thinking, thank you.

Hubert Nowotny August 30, 2017 at 10:32 am • Reply

Last-but-one scan:
[Aug 28 18:36:40] Scan Complete. Scanned 5492 files, 21 plugins, 1 themes, 35 pages, 3 comments and 26529 records in 15 minutes.
After that: Removed the WooCommerce Plugin (not yet in use), and cleaned up a bit.
Last Scan (with advanced technology):
[Aug 30 18:25:35] Scan Complete. Scanned 4266 files, 21 plugins, 1 themes, 35 pages, 3 comments and 22905 records in 2 minutes 6 seconds.
And it makes the scan run through at first attempt (which was not the case beforehand).
From 15' down to 2'06'' (with approx. 80% of files and records):
Simply great.
Thanks a lot!

Hubert Nowotny

Mark Maunder August 30, 2017 at 11:51 am • Reply

That's great data. Thanks for sharing Hubert.

Sara August 30, 2017 at 10:46 am • Reply

I've noticed a increase in the speed of the scans - truly a great improvement!!
Hats off once again, to the chefs in the geek department.

John Nadeau August 30, 2017 at 1:07 pm • Reply

Nicely Done!

Question: Will this use less, or more, CPU Resources on our Webhost?

Mark Maunder August 30, 2017 at 2:29 pm • Reply

Way, way less.

ron August 30, 2017 at 5:32 pm • Reply

I think the Wordpress team needs to better lock down sensitive files such as the wp-config.php, which should reside away from the root. And secondly, plugins which are abandoned or have not seen updates for years should be removed from the Wordpress repository.

Congrats on your great work and contribution.

Ron

Nyssa The Hobbit August 30, 2017 at 9:00 pm • Reply

That explains the sudden jump from 7-minute (and increasing) scans to 1-minute.... :)

Martin August 31, 2017 at 2:59 am • Reply

Thank you for the continued updates on CyberSecurity your information and help go a long way in assisting site admins.
I use your Wordfence plugin on all my websites and encourage all my clients to do the same.

I notice that new Wordpress installs come with a selection of plugins already loaded and I am surprised that they have as yet not made Wordfence one of them!

Get the latest WordPress security updates and news

Sign up for WordPress security alerts, Wordfence product updates and security news via email.