Go back

Show AI crawlers what you want them to see – with Arnout Hellemans

14m 58s

Show AI crawlers what you want them to see – with Arnout Hellemans

The discussion emphasizes the critical difference between raw HTML and rendered HTML for SEO, especially with the rise of JavaScript frameworks and AI crawlers. Arnaud, a tech SEO consultant, explains that while Google and Bing render pages, many AI crawlers (like OpenAI and Perplexity) only parse raw HTML due to high computational costs. This means content, headings, page titles, and structured data inserted via JavaScript may be invisible to these crawlers, harming discoverability. The key takeaway is to ensure the raw HTML contains all essential elements, aiming for parity with the rendered version. Tools like the View Rendered Source Chrome extension and Sidebulb crawler help diagnose discrepancies. Priorities should focus on core content (headings, titles, images) and structured data (e.g., review snippets), as missing these can directly impact rankings and CTR. For websites built on headless CMSs (e.g., React-based), pre-rendering solutions are necessary, whereas traditional CMSs (WordPress, Wix) are less prone to such issues. Arnaud advises testing changes and using data (e.g., CTR differences) to convince stakeholders of the importance. Ultimately, monitoring is crucial because JavaScript-based setups can break unexpectedly, requiring ongoing attention to maintain SEO performance.

Transcription

2117 Words, 11556 Characters

English
You should not only look at the source of the page, but also look at the rendered version. And as I said, a few rendered sources, the extension by John Horb is awesome for that. Side bulb has a great comparison in their crawler, and more people should just check this. I see a lot of people not checking this and thus failing to solve some issues. Hi, I'm Arnaud and this is a Theo in 2026. All right, what's your number on As you said before, 2026? I feel like people really need to look at their technical setup. And with that, I mean the rendered version versus the raw HTML, especially with a lot of AI crawlers not rendering yet, but also because I've seen a lot of this crap that's been seized in that part. I just see so is the rendered version typically quite different for different sort of engines? Well, as with a lot of things in the Theo, it depends, right? Sometimes when websites are being built using JavaScript frameworks, the content is actually different in the raw HTML versus the rendered HTML. So titles might be different, there might not be schema markup, there might be different headings, there might be whatever, because those are being changed by the execution of JavaScript. And that can severely impact the discoverability of your page. So what does this mean then for the use of JavaScript over the next few years? Well, you might see it this way, right? Crawling a website using rendered version, it takes a lot of energy from a browser. So you actually need to render the pages. So most would rather just scrape the page, get the raw HTML and get all the elements. And I think currently this is why a lot of the AI crawlers open AI, perplexity, etc., etc., it's just too expensive for them to do it. Microsoft with Bing and Google are doing it. The thing is, you can properly implement this, right? With pre-rendered solutions, with like hybrid solutions. So there's loads of ways to work around it. So it's not the death of the JavaScript, but I feel JavaScript has added a lot of technical difficulties that feels scary for most people in SEO. So most people don't even know how to do this, right? It took me a while to figure it out. So it's not definitely not the death because JavaScript has given us loads, interactive elements and all kinds of things. But it's something people should be aware of, especially with all these new crawlers popping up, right? And you want the content to be seen by AI crawlers as well as search engines, right? So what does Nessie need to do in order to determine how their website is seen by AI crawlers? So for one, there's a free extension in Chrome, which is view rendered source, which will show you the difference between the raw HTML and the view that the rendered source. The other thing is if you use a crawler, look at the difference between the raw HTML and the rendered HTML, I really enjoy using side bobe for that because it will literally show you these links were added using JavaScript or this piece of content has changed because of JavaScript or these images are being rendered through the execution of JavaScript, etc. etc. Got you okay, but you obviously say that AI search engines can see things quite differently to other search engines. You mentioned Microsoft, there is as well as Google. So how do you determine how they specifically see things differently and what do you do about it to ensure that they have a better idea of what's on your web pages? Well, basically, they only look at the raw HTML. So what you see in the raw HTML is what they can see. So if your images are invisible in there, then they want to see the images. If your page title isn't filled in or there's no headings in the raw HTML, then that's what they are seeing. So does that need to be corrected then quite simply? Yeah, yeah. In an ideal situation, the raw HTML and the rendered HTML is basically the same, right? Most of the fastest your web page will be a lot faster because there is no JavaScript execution needed to render the page in an ideal situation. Is that always possible? No, but you should get as close as possible because it makes everything faster and there won't be any annexation problems. It's way easier. Like back into days, we would just look at the raw HTML and that was it. No, it is, it's a lot more difficult. So how does an SEO prioritize what needs to be done? If they're working on a big website, hundreds of thousands of pages and they've got an issue with the non-rendered version and not everything obviously coming through the raw HTML, do you prioritize certain elements that should be incorporated or do you prioritize certain pages? Yeah. You have first for sure. The basics are the most important, right? The core content of the page can be read. The dismeans the headings, page title, your images, etc. Because that's what you want in the index. That certain elements are not working or for instance a footer is not working and like that's not the biggest issue. So you need to prioritize on having all the elements in the raw HTML and they should not change, right? What I see a lot or a half-see is that when the page gets rendered, the elements are still the same but they've been taken away and inserted again, which gives search engines like what's happening here. Because I thought I had the H1 and now the H1 is somewhere else. But still the same H1. If you have a JavaScript menu, do you need to prioritize the inclusion of HTML links to all the other pages on your site or can it be sufficient to actually just using an XML site map for that? Well, an XML site map is always a good idea but the challenge with links in these mega menus is when you switch of JavaScript, try doing it, right? Switch of JavaScript and your browser using a no script plugin or whatever and then try browsing your own website might be very hard. Yeah. So in an ideal situation, you want the internal linking to keep working but the biggest priority is getting the content and the right more copes, so the heading, the page title and your content in the index. Like that's the most important part. And does this all impact brand visibility? Obviously there are different features in the SERP. There are elements within AI search engine results that give brands a greater prominence. If you have certain elements incorporated with the HTML, I guess it gives you that increased brand visibility. Well, yeah, but both search engines will first crawl and index the raw HTML and then render the page and then compare it to the original and then say, hey, should I overwrite this? But for instance, say you would use JavaScript to insert a structure data that might be an issue because there might be a delay of the structure data being rendered. So for instance, a product might be out of stock. But now it's in stock when your website, but because the page hasn't been rendered, you will still think it's out of stock. So that's why those particular elements are really important to have in your raw HTML, not your rendered or not only in your rendered. How do you know which elements are likely to have the biggest impact on rankings? I mean, for instance, you obviously talked about basic HTML markup. I think again, it's a bit of an it depends, right? So say you would have a review stars. If you're using a JavaScript to insert that part of the code, it might not appear for all the pages immediately, right? So that is an important element. But say you would have like headings changing. So there's no headings in the raw HTML, but there is in the rendered. Now that will impact the ranking of a new article. Eventually it will fix itself once the indexed version or the rendered version gets indexed. But initially you won't benefit from the work you did. And I guess this conversation obviously emphasized these importance of including these elements and selecting a new CMS. If you're looking to hopefully create the next version of your website, then you've got to make sure that it incorporates the non-rendered webpage version as readable by very search engines, modern search engines. Yeah, and I see a lot of headless CMSs popping up, like Storyblock, et cetera, et cetera. And they usually have a JavaScript base app on the. And if you don't use a pre-rendered solution, so basically a server version that will render the page and serve to search engines and uses a fully rendered page. If you don't have anything like that, it will impact everything. Is it a CMS that you're particularly fond of or a CMS that you'd like to shout out is not doing a good job? I think any CMS can be adjusted to be able to do to serve this, right? It's not that it's a little more work, but it can be done. Most of the traditional CMSs use less JavaScript, which basically. Well, the core elements will just be there. You should just be aware, right? And I feel a lot of people are not aware. They say, I see a lot of use cases for react-based applications or websites, but there's also a lot of cases where you shouldn't. If you're making loads of changes, you don't need a developer for everything. You better go for a fairly more standard CMS out of the box, whether it's WordPress, Drupal, or whatever, or one of those builders, like Wix or Doodhour, whatever. Then building something with the react-frontend, because yeah, that will create these problems. But I think we should be aware that this is happening. And I see a lot of people that are a little scared of doing SEO this way. And obviously, it can be a lot of work to do this as well. So, do you actually implement the changes on some kind of test and learn basis to analyze the impact of what you're having? And also, if you're going to do that, how do you have that conversation with stakeholders internally to demonstrate the impact of what you're doing? I think you first need to understand what's happening. And you use tools like Fiorandard Source, but I've had projects where it would also be dependent on user agent, what IP you were in, what country. So, it's depth-cult, but then I think once you've understood what the problem is, you can build a case around it fairly easily. Because, say, for instance, the structure data is the review snippet is gone. For some pages and other pages it's still in. You can basically say, well, the CTR of these pages, the click-through rate, is a lot higher than for these pages. So, it's highly likely us to do with this. About you. So, tonight's what's the key takeaway from the tip you shared today? That you should not only look at the source of the page, but also look at the Fiorandard version. And as I said, a few rendered source, the extension by John Horb is awesome for that. And how much of an SEO's time should we spend on doing this? I guess it depends, but if you are going to work on a React-based platform, like a lot of time to fix this and monitor it. Because sometimes it breaks, and then you get, like, unforeseen errors, like, for instance, the pre-rendering stops failing or stops working, then you have a big problem, and most people aren't looking at that. It depends on what kind of platform. If you don't use any JavaScript in the front end or her BME, it's less of an issue, right? So, again, it depends. Our night's elements is a freelance tech SEO and analytics consultant find out more over at [email protected]. Our night, thanks so much for being part of SEO in 2026. Oh, good. Thank you for having me. I've been your host David Bane. Get your copy of SEO in 2026, the book over at SEO in 2026.com. [BLANK_AUDIO]

Podcast Summary

Key Points:

  1. SEOs must check both raw HTML and rendered source, as JavaScript frameworks often change content (titles, headings, schema) after rendering.
  2. AI crawlers (e.g., OpenAI, Perplexity) typically only read raw HTML due to high rendering costs, while Google and Bing render pages.
  3. Prioritize ensuring core content (headings, page titles, images, structured data) is present in raw HTML; less critical elements like footers can be lower priority.
  4. JavaScript-based menus should include HTML links for internal linking, though XML sitemaps help; the main goal is getting content indexed.
  5. Use tools like the View Rendered Source Chrome extension (by John Horb) or Sidebulb crawler to compare raw vs. rendered HTML and identify issues.
  6. Headless CMSs (e.g., Storyblok) often require pre-rendering solutions to avoid SEO problems; traditional CMSs (WordPress, Wix) are simpler for non-developers.
  7. Test changes and build stakeholder cases by showing impact, e.g., missing review snippets lowering CTR vs. pages with them.

Summary:

The discussion emphasizes the critical difference between raw HTML and rendered HTML for SEO, especially with the rise of JavaScript frameworks and AI crawlers. Arnaud, a tech SEO consultant, explains that while Google and Bing render pages, many AI crawlers (like OpenAI and Perplexity) only parse raw HTML due to high computational costs. This means content, headings, page titles, and structured data inserted via JavaScript may be invisible to these crawlers, harming discoverability.

The key takeaway is to ensure the raw HTML contains all essential elements, aiming for parity with the rendered version. Tools like the View Rendered Source Chrome extension and Sidebulb crawler help diagnose discrepancies. , review snippets), as missing these can directly impact rankings and CTR.

, React-based), pre-rendering solutions are necessary, whereas traditional CMSs (WordPress, Wix) are less prone to such issues. , CTR differences) to convince stakeholders of the importance. Ultimately, monitoring is crucial because JavaScript-based setups can break unexpectedly, requiring ongoing attention to maintain SEO performance.

FAQs

JavaScript frameworks can change content like titles, headings, and schema in the rendered HTML compared to raw HTML. This can impact discoverability by search engines and AI crawlers.

AI crawlers often only look at raw HTML because rendering pages is expensive, while Google and Bing use rendered versions. This means content added by JavaScript may be invisible to AI crawlers.

The 'View Rendered Source' Chrome extension by John Horb shows differences. Sidebulb's crawler also highlights changes like links or content added by JavaScript.

Prioritize core content like headings, page titles, images, and schema markup in raw HTML. Less critical elements like footers can be addressed later.

Yes, structured data inserted via JavaScript may have a delay in rendering, causing issues like outdated product stock info. It's best to include it in raw HTML.

Show examples where missing rendered elements (e.g., review snippets) reduce click-through rates. Compare affected and unaffected pages to build a case.

Chat with AI

Loading...

Pro features

Go deeper with this episode

Unlock creator-grade tools that turn any transcript into show notes and subtitle files.