Using Botify and Neural Networks to Write Meta Descriptions

Meta Descriptions. They’re not considered a ranking factor. Copywriters are not thrilled with the task of writing them. There’s lots of them to write. And if you don’t bother writing them, Google will just simply write them for you. So why bother?

Because they offer performance. Performance in the guise of a higher click through rate. If you write good meta descriptions, there’s promise of a higher Click Through Rate (CTR), which means more organic traffic. In fact best practice dictates that we should write descriptions just within a 155 character limit and include a call to action, because that is what Google will show in the search engine results pages (SERP). So that’s the motivation, what about the means? 

Say hello to neural networks

Deep learning is a form of neural networks, which is a subset of machine learning. They can be used to learn from data by modeling it and then having reached a level of statistical confidence, make reliable predictions or forecasts in the face of unseen data. 

The inputs of the neural network are translated into matrices, think 2 dimensional grids of numbers and then using linear algebra, the neural network uses a bunch of maths to learn from the numbers, spot patterns, form a model and then output.

The concept is rather abstract, however the above is the computerized version of showing a human enough examples of something to work out how to distinguish between objects or perform tasks.

In the context of generating meta descriptions, the neural network needs to see enough examples of meta descriptions to know how meta descriptions are written in order to produce their own.

Humans come preloaded with knowledge, for example a toddler would not need millions of images of cats and dogs to be able to tell the difference. Nor would a human copywriter need millions of meta descriptions to know to write one. Machines on the hand start naive and therefore need training data which would be millions of examples. 

With Botify, yes we can!

For a project of this scale and complexity, you need data and millions of it. Just ask the folks at OPEN AI, the creators of the GPT 3 algorithm that’s able to produce machine generated content such as meta titles and descriptions. They trained their neural network on over a billion documents.

Thankfully with the help of Botify, we managed to train our models on millions of web pages. Botify was critical to the project because most enterprise brands use them and therefore we were able to crawl enterprise sites in order to get the data (and enough of it) that was essential to training the model. They also have an API which makes data extraction a breeze, see our SEO API user guide.

When you have millions of web content and their associated meta descriptions to train on, a desktop application will not cut it. And we’ve tried in vain previously.

The inputs to our model were the body content, headings (both H1 And H2). Once we had the data, we also used machine learning to classify the page content type between product items, category, B2B landing pages. This was crucial as it would help the model pursue the correct approach to writing the meta description.

All of this would be translated into matrices, which would help the neural network understand the position of each letter to another letter, a word’s position to another word, word lengths, why certain words were combined in certain ways and not in others and so on. Effectively, we were training a machine to learn web page English for meta descriptions!

Was it any good?

The best judge is ultimately the Click Through Rate (CTR), in any case let’s see how the machine generated versions fared:

Human: Toolbox by Vitra
Machine: Shop for Toolbox by Vitra at John Lewis & Partners. Free UK mainland delivery when you spend £50 and over.

Human: Modern and elegant, Silk is a classic lighting series of hand-blown opal glass pendants, table lamps and floor lamps which are suspended on a delicate yet architectural brass stem.<p>The Silk pendant collection makes a dramatic impact when hung in a casca
Machine: Buy Silk Pendant Light With Similar products also available. SALE now on!

Human: A collection of rich brown decorations, the perfect finishing touch to your tree this festive season.
Machine: Brown decoration collection at John Lewis & Partners. Free UK mainland delivery when you spend £50 and over.

Human: This 200 Thread count woven cotton bed linen range has been washed to give a soft feel and relaxed look. New for AW18 is our charcoal colour way, ideal to add a sense of sophistication and luxury when paired with our rich velvet bedspreads. Featuring butt
Machine: Washed Cotton Fitted sheet Charms & Ingredients Ingredients. Similar products also available. SALE now on!

Human: Striking the balance between a traditional and contemporary Christmas, this stunning range of decorations are beautifully crafted from glass by the Original Winter’s Tale. Whether adorning your tree or placed throughout your home, create an elegant festiv
Machine: Buy Star Midnight Bauble Decorations from our Christmas range at John Lewis & Partners. Free UK mainland delivery when you spend £50 and over.

Personally, I think the results are very good (yes I would think so I suppose, however I’m also highly self critical!), when I consider how a few million data points is quite paltry compared to billions of web pages. Naturally, it could always be better (more grammatically correct for example, no repeated words) and we wouldn’t dream of charging our clients for any unusable auto generated meta descriptions.

What auto generated metas mean for SEO

The Google position is very clear on auto generated content, in that it is against it and therefore against the Google guidelines. However, we must also keep in mind that these were written before the (about to be) widespread use of Machine Learning in SEO. The world is changing, and we believe Google’s Guidelines will eventually change too. It may allow for auto generated content, however it may also increase the thresholds to punish bad grammar and spelling mistakes if it does not do so already.

Another consideration to keep in mind is that the motivation of Google’s Guidelines is to prevent manipulation of search rankings and not help users. Well doesn’t mean we should use SEO tools such as Botify for the benefit of users. Some would argue that the use of neural networks is not to manipulate the SERPs but to help users by having a credible meta description rather than none at all.

Afterall, the use of neural networks means that brands and agencies can benefit from consistently satisfactory if not well written meta descriptions because machines don’t get tired and don’t care about meta descriptions being beneath them as a copywriter. Machines will also learn from a consistent theme which will save resources by allowing writers to focus on more creative writing, something machines have yet to master.

Given neural networks are there to help users with a meta description (isn’t that what SEO is about?), I think it can be agreed that auto generated content technically violates the guidelines, however not the spirit as this isn’t a SERP manipulation exercise.

Can we apply neural networks to SEO content?

I think the answer is a firm yes, GPT 3 has already shown it’s already here and ready for testing. The above results show that even with our own ARTIOS algorithm and Botify’s data, with more training on client cases, we’re in a position to auto generate sensible meta description. And this can be extended to Titles, 1st paragraphs of body copy and paid search ads. However, you’ll need a lot of data and thanks to Botify, you now have the means to do so.

This article was contributed by Andreas Voniatis of Artios, helping companies profit from SEO automation.