Why Isn’t Google Indexing Some of My Site’s Pages?

There are numerous reasons Google might not index all of your web pages, and it isn’t always a difficult task.

Video Transcript

Dave: So, the problem that we’ve got is on our .ca site for Vernon Tarps. You can keep it recording. It’s fine. We have a whole bunch of pages where it says the page isn’t indexed. It’s crawled, currently not indexed, but there’s no indication what to do about it. And typically, what it is, David is products that don’t have a lot of content. But there was one. I think the example I gave you has a lot of content.

David: Yeah.

Dave: It doesn’t say it’s duplicate or anything like that. So, I’m just not sure what to do about it.

David: Okay, great. No, this is a super question. So, first of all, kudos to you for looking in search console and seeing this error. Right? Hooray. Second of all, when I say error, the word I should use is notice because this is not necessarily an error. This is Google saying, “For your information, we crawled it, but we decided not to add it to the index.” So, in some cases, that might be pages like your blog archive pages.

Dave: Sure. Those I don’t worry about.

David: Yeah, exactly. So, when you see them listed, this is not a list of errors. This is a list I need to go through to decide whether I want Google to index or not. Right? And so, in this case, this is a product page for a unique product. And yes, you do want Google to index it, right?

Dave: Right.

David: That is the use of this report. You’ve not only looked in search console, but you’ve also used the report properly. So, kudos. Two, there are many reasons why it may not be indexed, and it’s not just duplicate content. Okay?

Dave: Okay.

David: Some of it could be too deep in your website’s crawl. So, for instance, if the depth is measured from the homepage, now we know as SEO people, not everybody lands on the homepage. We want them to land on the most relevant page and start there. We don’t want them to have to find it. But for depth of pro, we measure from the homepage. So, how many steps, at the least, would it take to get to that page from the homepage? Homepage, some sort of category, maybe some sort of subcategory, maybe six pages deep on that subcategory to find the product. Well, that’s pretty far deep into the site. Because of that, Google may say this must not be a very important page because it’s just buried. This is very common with e-commerce sites like this one.

Dave: I’m wondering if that’s the case.

David: Yeah. So, you’re right to look at duplicate content because that would be another reason Google may not add it to the index.

Dave: But wouldn’t they say that?

David: No, they do not mention duplicate content in search console.

Dave: They have before. I’ve seen it.

David: I don’t believe so.

Dave: They did. Remember when we were doing the .ca and the .com at the very beginning?

David: Oh, oh, okay. That’s not the last time I’ll be wrong today.

Dave: Maybe it’s rare. Maybe it’s rare.

David: I think that’s the case. So, number one, if this is a very important product, find a way to move it up in the order. So, for instance, if the example I gave is six pages deep on a subcategory, then move it to at least the first page of that subcategory; maybe you could reorder it.

Dave: As I’m thinking about it, I think there are a bunch of pages that might be like this where there’s no reason why it can’t move up.

David: So that’s one way. Another way, and this is a really good reason for a blog, because blogs can get people to a particular product or deeper into a site a lot quicker. It’s that blogs are not just about ranking and SEO. They’re not just about long-tail traffic. The biggest value of a blog post is the internal links you can build to your other stuff, especially something like this, where it’s really deep. So maybe you just wrote a new blog post about things to consider when buying this kind of product and then link to a page that has some of those products. Now, that’s not as deep. Right? Now, when we get to blogs, if you’ve written in blogs long enough, those blog posts get really deep, right? You know, that’s where category pages come in. Category pages allow you to bypass the main blog archive and say, okay, I’m going to skip all the blogs and talk about all the blogs in one category, which then makes the shorter distance to the blog post. So, you see what we’re doing here? We’re finding ways to build internal links to something that’s important because Google uses internal links within your site to evaluate how important something is. If there are no internal links except one, it says, oh, that must not be all that important to them. So, I’m not going to bother to add it to my index because index is a limited commodity, right? They can only add so much. They don’t want to add everything. Now, the other side of it is thinking about this from an internal linking perspective, which could be helpful, too. So, this is why breadcrumbs are important, right? Breadcrumbs add internal links, but also anywhere you have content. We’ve talked about the importance of content on category pages for e-commerce websites, right? Well, one category of things is probably related to another category, and you might mention another category on a page just because it’s natural, right? Well, what if that became an internal link, too, right? So, this is one of these wonderful uses of Screaming Frog. And I’m a big fan of Screaming Frog. I’m big on not buying SEO tools before you need to. Right? I’m not here to sell you into more tools, but if you’re a developer and you have a very large website, Screaming Frog is a worthwhile investment because you can crawl a site and you can use it to find internal linking opportunities. There are several articles about how to do this. I wrote one on Curious Ants. There are some other recent developments in Screaming Frog that will help you crawl the site and find places where something, a variation of that idea, is mentioned. And if you add the links, now that’s a tedious process, going in and manually adding links. But in this case, if this product is really important to the client and not indexed because they found it but not indexed it, then a couple of internal links to make it no longer quite as deep might be just what you need.

Dave: Well, it still is the same level from the homepage, but maybe not from what you’re saying. It’s not from the blog itself.

David: Well, having more links is always going to help you. But presumably, some of those links are higher up in the architecture of your site. Right?

Dave: Right.

David: And therefore, if you put a link higher up in the architecture, it might not be twelve steps deep into your website, but now it might be six steps deep. Right? And it might be just enough. Now, six steps is still pretty deep into a website. So, then you just have to make a decision. All right, is this that important? Now let’s say clients, we have a special on this product, we really need to promote it. And it’s crawled but not indexed. Well, then, I’d simply add a link from the homepage straight to that product. And now the depth is one. You’re using the power of your homepage to build a link to that product, and that will probably be enough to get that index.