RafalAI Contact

ยท Study

Do AI models know this year's SEO changes? We checked six against the official docs

TL;DR. We asked six AI models 65 SEO questions with no web search and checked each answer against an official Google, Bing or OpenAI page. On long-established facts they got 30 to 32 of 32 right. On things documented or changed in 2026 they got 17 to 27 of 33, and their most common mistake was saying a real new feature doesn't exist. They made the same kind of mistake on general facts that change, so we can't say they're worse at SEO. Check anything recent at the source.

Why we ran this

Plenty of site owners ask an AI tool before they open Google's documentation. That's fine for things that haven't moved in years. 2026 moved a lot, though. FAQ rich results went away, Search Console got an AI opt-out, and OpenAI documented a new crawler. Do the models know?

What we asked and how we checked

We wrote 65 short SEO questions. 32 cover facts documented for years. 33 cover features Google, Bing or OpenAI documented or changed in 2026. Most are yes/no, and the rest have four options. For each one we saved the official page and the sentence that answers it, then re-checked all 65 against the live pages on October 26, 2026.

We asked six models through their APIs, answering from memory with no web search. We don't rank them, because a test this size can't separate them.

To see whether any gap was about SEO or just about recent facts, we added a control from FreshQA, a public set of general-knowledge questions. We took 40 questions whose answers never change and 40 whose answers change slowly, and re-checked every slow-changing answer against a current source first. We asked each question twice, once with the right answer and once with a plausible wrong one, and the model said whether the proposed answer was correct. Same day, same models.

What we found

Question set Items Example Correct, lowest to highest of six models
SEO facts documented for years 32 questions Does Google support the crawl-delay field in robots.txt? 30 to 32 of 32 (94% to 100%)
SEO features documented or changed in 2026 33 questions Does OpenAI document a crawler named OAI-AdsBot? 17 to 27 of 33 (52% to 82%)
General facts that never change (FreshQA) 80 judgments (40 questions x 2) Which country won the 2017 Eurovision Song Contest? 56 to 67 of 80 (70% to 84%)
General facts that change slowly, re-checked Oct 2026 (FreshQA) 80 judgments (40 questions x 2) How many member states are there in the Eurozone? 49 to 62 of 80 (61% to 78%)

Compare within each pair of rows, not across them, because the SEO and FreshQA questions use different formats.

Guessing baseline: a model picking at random would score about 40% on the established SEO row, about 41% on the 2026 SEO row and 50% on both FreshQA rows. The math: established SEO has 19 yes/no questions (1 in 2 chance each) and 13 four-option questions, including 2 that ask for a number (1 in 4 each), so (19 x 0.5 + 13 x 0.25) / 32 = 12.75 / 32 = 39.8%. The 2026 SEO set has 21 yes/no and 12 four-option, including 1 number question, so (21 x 0.5 + 12 x 0.25) / 33 = 13.5 / 33 = 40.9%. Every FreshQA judgment is yes/no, so 0.5.

The SEO gap has a clear shape. For 16 of the 2026 yes/no questions, the right answer is yes. Five of the six models said no on at least 8 of those 16. On the 16 matching long-established questions, they said no on 0 to 2.

The same model often answered the same fact both ways. Ask "does this exist?" and it says no. Offer it as one of four options and it picks it.

Each API also reports a probability for the answer it gives. On wrong answers, the average was 64% to 81%. On right answers it was 81% to 90%. That's too close to use as a warning sign.

Asking 30 questions a second time changed nothing. All 180 repeat answers matched the first ones.

Five 2026 changes the models missed

Search Console has an AI opt-out

Does Search Console have a setting to keep a site out of AI Overviews and AI Mode? 5 of 6 models said no.

It does. Google's help page says to "use the Search generative AI control to manage your site's inclusion" in generative AI features on Google Search, and lists AI Overviews and AI Mode. Google says the control "isn't used as a ranking or inclusion signal affecting other parts of Search."

When we asked where a site owner can opt out of these features while staying in Search, with four Google products as options, all 6 picked Search Console. Opting out of AI Overviews and AI Mode is a Search Console setting. Search Console Help.

FAQ rich results are gone, depending on how you ask

Does Google Search show FAQ rich results? 4 of 6 models said yes.

It doesn't. Google's updates page says "this feature will no longer appear in Google Search starting May 7, 2026."

Then we asked what the status of FAQ rich results is, with four options. All 6 picked "They no longer appear in Google Search," including the 4 that had just said yes. The models had the fact. Their answer depended on the question's format. FAQPage markup no longer earns a rich result, so don't spend time on it for that reason. Google Search Central updates.

Preferred sources isn't English-only

Which languages does Google's preferred sources feature cover for Top Stories? All 6 models got it wrong. Five picked "English only" and one picked "English and Hindi only."

Google's documentation says it's available "in all languages where Google Search is available."

On the yes/no version, "Is Google's preferred sources feature available only in English?", 4 of the 6 correctly said no. All 4 then picked a limited language list on the multiple-choice version. Google Search Central, preferred sources.

OpenAI has a separate ads crawler

Does OpenAI document a crawler named OAI-AdsBot? 5 of 6 models said no.

OpenAI's bots page says "OAI-AdsBot is used to validate the safety of web pages submitted as ads on ChatGPT." Its advertiser guidance says "You must allow OAI-AdsBot."

Asked which OpenAI crawler must be allowed for ChatGPT ad landing page review, 4 of 6 picked OAI-AdsBot and 2 picked GPTBot. Three of the models that said OAI-AdsBot isn't documented picked it here. Each OpenAI user agent gets its own robots.txt decision. OpenAI crawlers and OpenAI advertiser guidance.

Social accounts can go in Search Console

Can Instagram, TikTok, X and YouTube accounts be added as Search Console properties? 5 of 6 models said no.

They can. Google's July 2026 post says platform properties let you "track how your social and video posts on Instagram, TikTok, X, and YouTube perform on Google Search, Discover, and Google News," and that they "are globally available to everyone." Two of the five models that said no still gave the right count when we asked how many platforms can be added. Google Search Central blog and Search Console Help.

Is this an SEO problem?

We can't say SEO knowledge is worse than general knowledge. The models went stale on general facts too.

On the slow-changing FreshQA questions, they rejected the correct current answer on 11 to 21 of 40. A few examples:

  • 5 of 6 models accepted 20 as the number of Eurozone members. It's 21 since Bulgaria joined on January 1, 2026 (ECB).
  • 5 of 6 accepted Croatia as the newest Schengen member. Bulgaria and Romania joined fully on January 1, 2025 (European Commission).
  • 4 of 6 rejected 630 as the number of seats in the German Bundestag. The current Bundestag has 630 seats (German Bundestag).

Even on facts that never change, they agreed with a plausible wrong answer on 8 to 14 of 40.

A direct comparison wasn't precise enough to settle it. The sets differ in format, difficulty and recency. Every 2026 SEO question is about this year, while only 3 of the 40 slow-changing FreshQA questions changed in 2026. Our reading is that these models lag on recent facts in general, and this year's SEO changes are one place it shows.

What to do with this

  • For anything that may have changed in the past year, use a tool with web search on and ask for the official URL. Open it and check the page says what the tool claims.
  • Treat "that doesn't exist" as a reason to check. It was the most common wrong answer in our test.
  • Don't treat a confident answer as a checked one. In our test, the probability each API gave its wrong answers was close to what it gave its right ones. Chat tools usually don't show that number at all, and we didn't measure how sure a chat tool sounds.
  • Ask the same thing two ways, as a yes/no question and as an open "what" or "which" question. If the answers disagree, trust neither.
  • Read the change logs yourself. Google's Search Central updates page and OpenAI's crawler page cover much of this year.

What this test can't tell you

  • It's small: 65 SEO questions, six models, one day.
  • The models answered from memory. Chat tools that search the web may do better, and we didn't test them.
  • We wrote the SEO questions. The 2026 ones are about newer, more niche features, so "new" and "obscure" are mixed together.
  • "Correct" means what the official page said on October 26, 2026. For some 2026 questions, the date of the change comes from news coverage.
  • We don't know when each model's training data ends.
  • Several questions cover the same feature, so there are fewer independent facts than questions.
  • FreshQA has been public since 2023, so some of it may be in the models' training data.

Run your own test

  1. Write 10 questions about SEO changes from the past 12 months, taken from official change logs and help pages. Save the URL and the sentence that answers each one.
  2. Write 10 matching questions about facts documented for years, in the same format.
  3. Add three or four made-up features whose answer is "no" to see whether your tool leans toward yes or no.
  4. Ask your key facts both as yes/no and as open questions.
  5. Add a control from FreshQA: about 10 never-changing and 10 slow-changing questions. Re-check every slow-changing answer against a current source and drop stale ones. In our sample, 8 of the 48 answers we could check had gone out of date within six months of the April 2026 release.
  6. Run everything in fresh chats, with web search off and then on. Note the tool, version and date.
  7. Score both sets. If your tool slips by a similar amount on recent SEO changes and slow-changing general facts, it's a recency problem, so use search and ask for links. Repeat after major tool updates.

About the test

None of these models is ours. All six are third-party models we called through their APIs. The six models were JEV 1.13, JEV 1.14, SubQ (snap-preview), Baseten mammoth and elephant, and OpenAI's gpt-6-luna. Each API returns a probability for every answer option, and we counted the option with the highest probability as the model's answer. Each question was asked on its own, with no web search, on October 26, 2026.

Sources

Tell us what your team does by hand, or what you want it to do. We'll reply with how we would build it.

Start a conversation