Why Are Some Pages Not Indexed by Google?

Here’s a brief intro to how Google indexes web pages.

Video Transcript

David: All right, so it sounds to me like you have seen in Search Console the number of pages indexed on a site decreasing and the number of non-indexed increasing.

Onawa: I have. And I, at one point, checked a couple of the links and had Google check them and reindex them. Eventually, I decided I needed to just leave it for a bit because I was getting these errors even though it said it was fine and indexable. So, I left it for a while, and it’s still going down a little bit.

David: When you said errors, what do you mean?

Onawa: Wherever you get the email that says Google…

David: A notification?

Onawa: Yeah.

David: Okay. What did the notification tell you about?

Onawa: Let me look. I have to go to a different email account.

David: Okay, everybody, how many email accounts do you have?

Dave: Oh, my gosh. I don’t have enough fingers and toes.

David: Right? Exactly. So, while you’re looking for that… First of all, I’m glad you’re looking in Search Console. That should be part of something we do every week for all of our clients. The Search Console report is just the best way to see how Google understands our content. And I’m glad you looked in there and saw it. When we look at that report in Search Console, to make you rest assured a little bit, sometimes, we don’t necessarily want every page indexed. Right? So, what we’re looking for in that report is if there are important pages not indexed. So, for instance, if it’s page six of the awesome sauce category blog posts, we don’t care if it’s indexed. Right? If it’s a blog post, then we should start paying attention to that. If it’s a landing page that sells services, then we absolutely need to panic, right? If it’s the privacy policy, who cares? Right? That’s just an internal page. So, once we determine that the pages in the not indexed report are ones that we really kind of don’t care about because not every page exists for SEO purposes, sometimes it’s explaining who we are or just in navigation, and that’s okay. But did you find the error that you saw?

Onawa: Okay, well, I had the email on one account, and I had to log into another account to look at Search Console. It said some fixes failed for page indexing issues on site.

David: Okay.

Onawa: And now, I’m going to the actual link. Okay, the link for more information just took me to validation failed, and the regular page indexing was not found.

David: Sure. So, let’s go to the Search Console account. Let’s go to the report that shows us pages that are not indexed. It will give you a list of all the pages that are indexed and potentially a graph showing you that’s increasing.

Onawa: The overall not indexed rather than the crawled currently not indexed. Which is more what I was trying to look at when I was messing with these. To try and have it reindex them or index them initially. That is going up, apparently. Maybe I’m just not looking at the right set of numbers.

David: Right, and crawl but not indexed is different than, as you say, there’s an index page and there’s crawl but not…. So, sometimes, Google Search Console says, hey, we know that’s existed, but we haven’t crawled it yet. And sometimes, if they’ve crawled it and determined not to index it based on the crawl. Okay. So, in the pages that it’s listing as, and this is not definitive, this is a sample. What are you noticing about those pages that are not indexed?

Onawa: It says crawled, currently not indexed. That’s trending up.

David: Right. So, there are more and more that have been crawled but are not being indexed. But when it lists the pages underneath that, what are you noticing about the pages listed?

Onawa: Some of them aren’t regular pages. Some of them end with like, feed, some version number…

David: Right. So, feed is the way WordPress works. Every page has a feed based on comments. And frankly, we kind of really don’t care if Google indexes the feed. You might be able to help by turning off the ability of the website to even have comments if you are not managing comments. And then you’ll get a bunch of 404 errors. But that’s okay because those pages no longer exist. And that’s what we want. We don’t want Google to even see them. So, the feed is okay. What are some of the others that are not feed-related? Like version numbers in it? It’s usually a JavaScript file or a CSS file.

Onawa: Yeah, that’s a JavaScript file.

David: Okay. Google doesn’t index JavaScript. I don’t care.

Dave: Would there be some image links as well?

David: Good question.

Onawa: There are tags and project types.

Dave: Google is probably smart enough to not even report on that.

David: Well, again, tags are a navigational thing. We can talk about the fact that sometimes websites go a little crazy when they start tagging blog posts and just tag on every other word they can think of, and they just create this confusing mess of tags that are totally not necessary. Frankly, if you’ve done that, I don’t blame Google for ignoring them. If I find the person who gave advice to SEO saying hey, you should start tagging your blog posts, and that helps your SEO, I’m going to punch them in the face. That’s the stupidest advice I’ve ever heard. Tags are a way to organize your content so people can find it easier and click to it, but it doesn’t help your SEO to tag posts. It just creates an unnecessary number of pages for Google to crawl, which, frankly, we don’t really want. So, it might be a problem where the tagging of posts is a little crazy. Or sometimes, you said portfolio pages?

Onawa: Yeah.

David: Is this a site that has a portfolio?

Onawa: Yeah.

David: Okay. Sometimes themes and stuff will have a portfolio component, but it’s not really used. Right? And so, at that point, you just got to turn it off in the back end if you’re not using it. But if you’re using it, well, then often portfolios are simply like images.

Onawa: This one’s got mostly images. It’s got a little text. The rest of what it’s not liking seems to be portfolio pages. They’re live.

David: So, oftentimes portfolio pages are… I don’t have any idea what client this is. Let’s say this might be a web development company, right? And a web development company might have a list of portfolios of all the sites they worked on. And so, they say, here’s a beautiful picture of the site we worked on for this company. Maybe they even link to the company. But that’s all the content on the page. There’s not enough content for Google to sink its teeth into. Now, if you wrote a 200-word description of not just the company but the process, and the problem you were trying to solve when you created this website, then it becomes a very helpful portfolio piece because it’s not like we’re going to just create a beautiful site, we solved a problem. And now, Google will have enough words. And frankly, users will have enough value on that page to index it.

Onawa: Is it worth doing that for each client? Not just in the sense of what we did to solve their problem, which is great, but what their business does. That’s what we have on there now. It’s like a little part about the business. Here’s a link.

David: Right. In that case, I don’t think it’s worth it because it’s just an internal navigation page. If people want to view a web development company’s site to, say, hey, I like the way these people work, and this is great. But you’re not necessarily going to get a new customer that way. If you blew it out and talked about it as a solution, and maybe it’s not even about the company, maybe it’s with the industry. Maybe it becomes a landing page about web development for plumbers or something like that. Right? Then it becomes helpful if you can show keyword research volume for these kinds of ideas. But ultimately, that page probably exists for users, not for search engines. And that Google says it’s not going to index. It doesn’t mean you suck. It means okay, only people who are on this web development company’s site or their portfolio, well great, humans can still navigate to it. It’s just not a page that’s going to be a landing page driving additional traffic.

Onawa: It is basically all tags and portfolio pages. I think I was trying to figure out why aren’t the portfolio pages indexed. But that makes sense.

David: Yeah, it’s kind of our SEO strategy. Remember, part of the process of Curious Ants that I’m recommending is that you create what we call an SEO blueprint, which is kind of a map of all the pages and the ideas that we’re focusing on for each of those pages. And not every page is going to have a keyword focus. For instance, your About Us page should be about your company. We’re not going to SEO the About Us page. Right? Plus, we have to remember, we’re not picking one or two words for every page. We’re picking a list of words per page. And so, sometimes, with old-school SEO, we used to pick a list of words that we wanted to rank for, and we’d put them amongst different pages. And so, we’d optimize the About Us page for a variation of the keyword we wanted. But Google is much more sophisticated than that now. They lump words together and use synonyms – kind of like we talked about last week when we saw the spelling variations, right? So, it isn’t worth it to build a page for every keyword or try to optimize every page for a different keyword. Think topically about the page, understanding there are many keywords. And so, we find that there may be some pages that we have no SEO use for. But they’re very, very important for helping someone decide to make a purchase, for instance. On my main site, I have a testimonials page. Right? And every time someone says something nice to me, I put it on there, right? Or I have a credentials page that shows my certifications. Here’s who I’ve written for, and here’s where I’ve spoken before. But I’m not really expecting to get traffic for that, but if someone is considering whether they want to work with me or not, I can see that they visit that page in that process before they even contact me. So, it’s not an SEO page, but I get conversions because of it. Right?

Onawa: Right.

David: And portfolio probably works the same way. It’s not an SEO page. But that’s okay. Are other pages? And creating the SEO blueprint to say, here’s where we’re focusing, also prevents us from getting distracted by this. Now, we could develop an SEO strategy to try to build more traffic to portfolio pages and possibly get more traffic and customers from it. But I think there are probably other opportunities that might pay off better.

Onawa: Okay.

David: But I like the way you thought about this because that’s how we think about these reports. The reports are telling us what Google is doing, but just because they’re doing something like not indexing a page… Okay, it makes sense to me that page is not indexed, right? It’s not Google saying you suck or your website is terrible. In fact, you could be doing everything right. Blogging as frequently as possible, writing really great and new additional landing pages. Well, as you do that, the Archives, for instance, or as you have successful websites built, your portfolio pages are going to be increasing. And therefore, the non-indexed pages are going to be increasing. But you’re doing everything right. And so, we just need to look and see if your main service area here, the I Build Interior Design Projects page, which is part of your SEO blueprint, is on that list. Then we figure that out. God forbid the home page appears on that list, which I have seen before, and it’s not really great. But good question.

Onawa: Well, that makes me feel better anyway.

David: Good. That’s what we’re all about – feeling better.