Is There a Way to Prevent Google from Scanning Certain Pages?

Are you curious about what Google should scan on your website? Let’s examine this.

Automated Transcript:

Christ to from lavanya. I’m using screaming frog to skin my website. That’s awesome. How can you tell Google not to scan certain pages, or is the cop current term indexing? For example, I’m seeing a disclaimer, cookie and privacy policy Etc. Come up when scanned. I’m trying to redo my site map for Google search since I changed a lot of pages. So so first of all, I’m really glad you’re using screaming frog. Screaming frog is one of my favorite tools. And for those of you who don’t know, is screaming frog is, it’s a crawler that you can use to crawl and visit every page of your website, that resembles the wig, Google might crawl your website. And so, as it crawls your website, it will tell you all kinds of information. Nation including hey you’ve referenced a page with page doesn’t exist anymore. Hey this title tag on this page is too long or Too Short. Hey you said have a canonical tag but you also have a no index tag on this page. That’s a contradiction. There’s all kinds of really wonderful things you can do with this. So I’m really glad lavanya you’re doing this. This is really good. You want to you ask? How can you tell Google not to scan certain pages? Do you mean how do you tell screaming frog? Not to Scranton not to crawl, certain pages or come out whatever I thought would ever scream if all pulled out, that’s the same thing at Google pulls up. Okay. Okay. That’s a okay. So first of all, let’s make a distinction screaming frog is a crawler like Google has a crawler, but it is not necessarily the same. However, If you’re finding a page with screaming frog, it’s likely that Google can find it as well. Hmm. And vice versa, if screaming frog doesn’t find a page, it might be that Google can’t find that page. So so if you’re so you’re asking about certain pages like your disclaimer, your cookie or privacy policy that you don’t necessarily want, Google to discover, but screaming frog is so use a suspect Google is as well. Uh-huh. Okay. So first of all, there is a really easy way to tell Google and screaming, frog, presumably not to crawl. A particular page in that is by adding the no index meta tag to the page if you’re using WordPress into, you’re using the yoast plugin. you can open up the page, go to advanced settings in Yost and say should it asks should Google be able to crawl this page and you say drop down know Okay, this does couple things. It adds the no index paytag to the page. But it also removes it from your XML sitemap. Automatically because Yost is generating your XML sitemap. So if you want to remove those pages, so Google does not crawl those pages. Then that was behavior for instance. I don’t want Google crawling my confirmation. Pages, whenever I see someone submits a form. Because theoretically, you only way someone should get to the confirmation page is by submitting form, and that’s why it’s set up as a goal. So, I go into that page and I go into advanced settings under the Yost section. And I set it to no index and then it removed from the XML sitemap and it, if Google does happen to find that, they all know not index that page, Now, one of the questions, I think there’s an implicit question behind what you’re asking, which is do you want Google to crawl, your disclaimer, your cookie page and your privacy policy. I thought we didn’t want them to do that. That’s why I asked. So do we want them to do that? That that’s it. Me. Yeah that’s an interesting question because in one sense, we were not going to get SEO traffic from those pages, right? It’s not like the purpose of the privacy policy is to get people searching for privacy policy to land on her page. Hmm. But if I say this is a big, if if Google believes that it’s credible for a website to have a privacy policy, And it can’t find it, then it wouldn’t be able to establish that’sa credibility Factor. That’s a huge if because I don’t I’ve never heard anybody say that. Google expects to see us privacy policy and if they do, they will give you a reward for that. But if we know index it and it is possible, then it would prevent Google for these Pages. Usually, what I do is just let them be. I don’t try to hide them from Google. But if Google doesn’t want to index him, it doesn’t hurt my feelings. Okay? So you might just leave them. Okay, my just leave me. It might not need any action and it let Google decide what it wants to do with those pages. And it may well ignore a lot of privacy policies are back. Exactly the same. And so, what does Google do when it discovers duplicate content ignores it. Right. So, we don’t really care if Google indexes our privacy policy that’s for people not for the Google bot. and and that would be okay or cookies policy, or our return policy, You know, maybe return policy, we do want to Google the next. In case someone says, how do I make a return on my wet on amazon.com? Maybe then we’d want Google to index that, but yeah. But that’s an interesting question. Thank you for asking. Okay, thank you.