Understanding Search and AI Chat Bots

Home » Fifth Wednesday Webinars » Understanding Search and AI Chat Bots

Do you want your website to appear in a Google search? Do you want your business recommended by an AI Chat? Your first step is to make sure their search bots can access your site.

Video Transcript:

So, is your digital garden still bearing fruit?

There’s a lot of talk out there about how much traffic people are losing to their websites. Or you may notice a drop in website traffic.

Or you may have experienced that you’re losing customers recently. Is all this because SEO is dead? Is it because AI is taking over?

We want to solve these problems by looking at our digital garden. And by the digital garden, I want to pull a metaphor here, I’m thinking about our website. It is a garden. It’s got all kinds of wonderful plants in it. These plants are the products and services you have to offer.

And, you know, you want these plants not just to be beautiful. You want them to produce fruit. You see, you can have a great, lovely garden, but if the garden isn’t producing food, you’re going to be hungry.

We don’t just want a beautiful website for people to look at; we want something that actually produces fruit: income, business, and growth.

That’s what we’re aiming for, even if it seems harder than it used to be.

Now, my wife has a garden at the local Greenway, so we’ve been playing it all summer, and she’s finally harvesting, you know, the final crop over the last couple of weeks. But one of the things, if you’ve ever grown vegetables, is you know that before there can be a vegetable, or actually more biologically, before you can have a fruit, you have to have a flower.

The same is true of your website. You can have a beautiful plant, but if you’re going to get fruit out of it, you need a flower — like a blog post, like a page that talks about what you have to offer. Content that’s engaging, for example, that’s kind of the flowers that can produce the fruit that is the business of your website.

And so, before we have any fruit, we need a flower. And flowers attract pollinators. Pollinators come from all over. It can be a butterfly, or it can be a bee. It’s going to pollinate the flower and produce the fruit.

Unless I’ve really overstretched my metaphor here, these pollinators are like the web crawlers, right? So, for Google to show you in search results, it first has to send a crawler (or pollinator). We’re pulling our metaphor here to look at your blog post and your fabulous flowers and say, “This is something of value that I could potentially serve to my customers.”

They visit your garden flowers, and they bloom. These crawlers (pollinators) gather data to send potential visitors to your site.

Now, crawlers make the fruit possible, right? But just because a web crawler, like a Googlebot, comes to your website, doesn’t mean it is going to produce fruit. And this is where we have to pause for a moment. Today, we’re going to talk a lot about web crawlers and how your website is discovered. But we also need to remember that the web crawler is one piece of this puzzle. You can’t just expect pollinators to come to your website or to your garden and produce fruit. Your plants need water, and you need fertilizer to help them grow. And the same thing’s true about your website. Just because the Googlebot visits your website doesn’t mean you’ll get any visitors. You have more work to do.

You have to produce new and engaging content. You have to establish yourself as an authority. You need a good, readable website for the Google crawler. So even though today we’re going to be talking about the crawlers that come to your website, which are the first step to having a fruitful website, let’s remember in the back of our minds that that’s just the first step. There’s a lot more you need for your garden —and for your website to grow.

So here we have our pollinators. They are all fluttering around the flowers of our garden, pollinating the fruits, helping us understand our garden so visitors can come and see our site and make a purchase, and helping us market the services we offer. But OMG, there are two new pollinators at the bottom of this page. These little spiders. They’re a little scary, aren’t they?

They’re new. They’re strange. We’re used to bees and butterflies, and suddenly these new crawlers. Well, in the metaphor we’re trying to pull, this is like having the new AI crawlers come to our website.

For Google to index our website and serve it to potential customers, the crawler has to visit it first. This is true with ChatGPT.

The ChatGPT bot is crawling the web, reading and understanding it so it can add your website’s information to its index and recommend you to people using it. It works the same way. Claude has a bot or three. Perplexity has two bots. These bots are out crawling the web, you know, and not to forget, like Bing has a bot in its own, right? They’re crawling your web pages to determine whether they’ll serve them to their users.

Well, some people are overreacting to these AI bots. They hear that, you know, these AI bots are coming to your site, and you may be turning them off. So they’re taking more proactive steps to prevent AI bots from accessing your website.

You know, we’re learning about things like CDNs, like Cloudflare. When you set up Cloudflare, it blocks AI bots by default. We’re learning that some web hosts, because these AI bots are pretty resource-intensive, are preventing AI bots from coming to your website.

Well, it might sound like that’s a good idea, but when you’re killing off the pollinators of your garden, bad things are going to happen: you’re not going to be able to produce fruit.

Now, there may be circumstances where you want to block an AI bot, like if you’re selling ads on your website, or if you’ve got a recipe blog and you don’t want AI bots stealing your recipes. And so there may be a good reason in those circumstances to block AI bots. However, you take that pretty radical step. In that case, you’d better have an excellent marketing plan to make up for the lost traffic that you’re going to have, because you’re not going to get traffic from the AI bots if you block AI bots from getting it.

And anybody who’s ever used the WordPress website and checked the box labeled “Discourage search engines from indexing the site” knows precisely what we’re talking about. You’re just doing the same thing as the checkbox does for Google bots: blocking AI bots. So the implications are the same. You will get no traffic. Without the traffic, you might have to reconsider your entire marketing strategy.

So there’s a new thing we have to do when it comes to website maintenance, especially when the website is trying to accomplish a goal like building a business. Because we need to keep track of what’s going on. We need to watch things like whether the pollinators are coming to our site. We need to ensure that these bots are visiting the different pages of our website.

There was recently a big lawsuit — I don’t know if you heard this —between Reddit and Perplexity. Because the Perplexity bot was forbidden. We need to crawl Reddit; however, it crawled Reddit anyway and used it to develop its AI systems and index. The other side of this is to make sure the bots you don’t want visiting are unable to or are not currently visiting.

This is something you’ll need to start paying attention to to consider AI as part of your marketing strategy. And there are all kinds of ways to keep track of the pollinators visiting your garden, the web crawlers visiting your website.

The gold standard is always a server log. Your web host keeps a log of everyone —every bot, every user agent — that visits your website.

If you have access to that, that’s a really valuable tool. Unfortunately, most web hosts don’t make it accessible to most website owners.

And so then we have to resort to other things. Now, Google Webmaster Tools is an excellent free tool that lets you see how Google interprets your website. And that will include the Google AI bots that feed into Gemini, AI mode, and AI overviews. If you want to keep track of Google bots visiting your site, Google Webmaster Tools is the best way to do so.

Don’t neglect Bing Webmaster Tools, too, right? Remember, Microsoft — just today, ChatGPT announced it’s going to become a for-profit company. And the most prominent owner in that is Microsoft. So ChatGPT has a strong relationship with Microsoft and its search engine, Bing. So we should be paying attention to how the Bing bot interacts with our websites, because that’s probably going to be a piece of the pie.

But that’s only for those two websites. We don’t have data from ChatGPT, Claude, or Perplexity to see how their tools are interacting with our website.

That’s why we built the WordPress crawler plugin and the WordPress crawler record plugin. And there’s the link right there. This is keeping track of how these crawlers — not just the AI bots, but the Google bots, the Bing bots, and even DuckDuckGo — access your web pages. The ones last time they came —what website, and which web pages of yours did they come to?

This is really helpful for WordPress because most WordPress hosts do not allow access to server logs, so you’d be running blind when it comes to looking at when the crawlers from the different AI tools come to your site, and that’s the reason why that would be a helpful plugin for you to consider.

But once we have a record of the pollinators—the crawlers from the different tools coming to our site—we’re going to want to make sure we crack more of them.

One of the interesting things I’ve seen while running the crawler record plugin on my websites for the last few months is how infrequently they actually visit some of my sites.

The most popular sites, they come more often. The sites that are like test sites and similar don’t visit very frequently.

That makes perfect sense. But the things that attract these pollinators, well, you know, things like having a good quality website that’s easy to read and fast.

The fact is that the new crawlers from the AI systems are far less sophisticated than Google’s crawler. Google’s crawler runs on Chromium, which is a very stripped-down version of the Chrome browser. And so it can do things and view parts of websites that an old bot — sorry, an old Google bot — can’t. And frankly, a lot of the new search engines, new AI bots can’t do either. So having a fast, crawlable website is going to help everybody. And so that, by making sure your technical SEO is excellent, these AI bots can read your site. You want to, obviously.

To do good SEO practice, good content that’s quality, you write that, is always going to be valuable to the bots and to the people that you go to. But one of the things we hear a lot about lately is the temptation to write AI content and publish it on our websites. And if human is the only thing that you’re worried about, that’s okay. However, AI bots don’t want to read AI-generated content to add to their AI index to answer questions. So if you want to be reached by Google’s AI or Claude, don’t use AI on your website for content.

As always, too, good SEO practice. What do other websites say about you? Links from those websites to you make a difference.

It not only helps your website establish credibility, but also helps those crawlers find your content on your site more easily.

As we discuss finding content on your website more easily, consider a tool called IndexNow. This is a tool that basically informs some of the search engines that something has been updated on your website. Bing uses its search engine to tell Bing that something has changed on a website. Several others think DuckDuckGo uses it as well. Google does not acknowledge that IndexNow is part of its indexing system. However, they have admitted they are watching it. They don’t use it. So it’s still valuable. Whether or not the AI bots use IndexNow, we don’t really know.

But again, the partnership between Microsoft and ChatGPT, for instance. They might be using some of the Bing Index to help them find content. So if you’re getting it quicker through an Index Now system, that might help ChatGPT find you a little bit better.

And just as always, good best practices on your website are always going to be good for you, for the search engines, for the AI, and for your users.

So even if none of the AI systems like you, your users are going to appreciate a fast, easy-to-use, easy-to-read, helpful content website.

So that’s the worst that’s going to happen. And so that’s how we attract pollinators to our website, always remembering that the pollinators are just the first step.

We need to follow all the best SEO practices if we’re going to take advantage of this. And that’s where Curious Ants comes in. Curious Ants is our little product. The ant is a pollinator, as a matter of fact. I didn’t think about that when I built the system.

But within Curious Ants, you can find not only free documentation on how to do everything to be seen by your customers, whether through search or AI.

There are also the weekly group coaching calls, where we ask and answer specific questions related to what we’re struggling with or questions we’ve encountered from our clients.