Optimizing AngularJS Single-Page Applications for Googlebot Crawlers

Posted by jrridley

It’s almost certain that you’ve encountered AngularJS on the web somewhere, even if you weren’t aware of it at the time. Here’s a list of just a few sites using Angular:

  • Upwork.com
  • Freelancer.com
  • Udemy.com
  • Youtube.com

Any of those look familiar? If so, it’s because AngularJS is taking over the Internet. There’s a good reason for that: Angular- and other React-style frameworks make for a better user and developer experience on a site. For background, AngularJS and ReactJS are part of a web design movement called single-page applications, or SPAs. While a traditional website loads each individual page as the user navigates the site, including calls to the server and cache, loading resources, and rendering the page, SPAs cut out much of the back-end activity by loading the entire site when a user first lands on a page. Instead of loading a new page each time you click on a link, the site dynamically updates a single HTML page as the user interacts with the site.

Image c/o Microsoft

Why is this movement taking over the Internet? With SPAs, users are treated to a screaming fast site through which they can navigate almost instantaneously, while developers have a template that allows them to customize, test, and optimize pages seamlessly and efficiently. AngularJS and ReactJS use advanced Javascript templates to render the site, which means the HTML/CSS page speed overhead is almost nothing. All site activity runs behind the scenes, out of view of the user.

Unfortunately, anyone who’s tried performing SEO on an Angular or React site knows that the site activity is hidden from more than just site visitors: it’s also hidden from web crawlers. Crawlers like Googlebot rely heavily on HTML/CSS data to render and interpret the content on a site. When that HTML content is hidden behind website scripts, crawlers have no website content to index and serve in search results.

Of course, Google claims they can crawl Javascript (and SEOs have tested and supported this claim), but even if that is true, Googlebot still struggles to crawl sites built on a SPA framework. One of the first issues we encountered when a client first approached us with an Angular site was that nothing beyond the homepage was appearing in the SERPs. ScreamingFrog crawls uncovered the homepage and a handful of other Javascript resources, and that was it.

SF Javascript.png

Another common issue is recording Google Analytics data. Think about it: Analytics data is tracked by recording pageviews every time a user navigates to a page. How can you track site analytics when there’s no HTML response to trigger a pageview?

After working with several clients on their SPA websites, we’ve developed a process for performing SEO on those sites. By using this process, we’ve not only enabled SPA sites to be indexed by search engines, but even to rank on the first page for keywords.

5-step solution to SEO for AngularJS

  1. Make a list of all pages on the site
  2. Install Prerender
  3. “Fetch as Google”
  4. Configure Analytics
  5. Recrawl the site

1) Make a list of all pages on your site

If this sounds like a long and tedious process, that’s because it definitely can be. For some sites, this will be as easy as exporting the XML sitemap for the site. For other sites, especially those with hundreds or thousands of pages, creating a comprehensive list of all the pages on the site can take hours or days. However, I cannot emphasize enough how helpful this step has been for us. Having an index of all pages on the site gives you a guide to reference and consult as you work on getting your site indexed. It’s almost impossible to predict every issue that you’re going to encounter with an SPA, and if you don’t have an all-inclusive list of content to reference throughout your SEO optimization, it’s highly likely you’ll leave some part of the site un-indexed by search engines inadvertently.

One solution that might enable you to streamline this process is to divide content into directories instead of individual pages. For example, if you know that you have a list of storeroom pages, include your /storeroom/ directory and make a note of how many pages that includes. Or if you have an e-commerce site, make a note of how many products you have in each shopping category and compile your list that way (though if you have an e-commerce site, I hope for your own sake you have a master list of products somewhere). Regardless of what you do to make this step less time-consuming, make sure you have a full list before continuing to step 2.

2) Install Prerender

Prerender is going to be your best friend when performing SEO for SPAs. Prerender is a service that will render your website in a virtual browser, then serve the static HTML content to web crawlers. From an SEO standpoint, this is as good of a solution as you can hope for: users still get the fast, dynamic SPA experience while search engine crawlers can identify indexable content for search results.

Prerender’s pricing varies based on the size of your site and the freshness of the cache served to Google. Smaller sites (up to 250 pages) can use Prerender for free, while larger sites (or sites that update constantly) may need to pay as much as $200+/month. However, having an indexable version of your site that enables you to attract customers through organic search is invaluable. This is where that list you compiled in step 1 comes into play: if you can prioritize what sections of your site need to be served to search engines, or with what frequency, you may be able to save a little bit of money each month while still achieving SEO progress.

3) “Fetch as Google”

Within Google Search Console is an incredibly useful feature called “Fetch as Google.” “Fetch as Google” allows you to enter a URL from your site and fetch it as Googlebot would during a crawl. “Fetch” returns the HTTP response from the page, which includes a full download of the page source code as Googlebot sees it. “Fetch and Render” will return the HTTP response and will also provide a screenshot of the page as Googlebot saw it and as a site visitor would see it.

This has powerful applications for AngularJS sites. Even with Prerender installed, you may find that Google is still only partially displaying your website, or it may be omitting key features of your site that are helpful to users. Plugging the URL into “Fetch as Google” will let you review how your site appears to search engines and what further steps you may need to take to optimize your keyword rankings. Additionally, after requesting a “Fetch” or “Fetch and Render,” you have the option to “Request Indexing” for that page, which can be handy catalyst for getting your site to appear in search results.

4) Configure Google Analytics (or Google Tag Manager)

As I mentioned above, SPAs can have serious trouble with recording Google Analytics data since they don’t track pageviews the way a standard website does. Instead of the traditional Google Analytics tracking code, you’ll need to install Analytics through some kind of alternative method.

One method that works well is to use the Angulartics plugin. Angulartics replaces standard pageview events with virtual pageview tracking, which tracks the entire user navigation across your application. Since SPAs dynamically load HTML content, these virtual pageviews are recorded based on user interactions with the site, which ultimately tracks the same user behavior as you would through traditional Analytics. Other people have found success using Google Tag Manager “History Change” triggers or other innovative methods, which are perfectly acceptable implementations. As long as your Google Analytics tracking records user interactions instead of conventional pageviews, your Analytics configuration should suffice.

5) Recrawl the site

After working through steps 1–4, you’re going to want to crawl the site yourself to find those errors that not even Googlebot was anticipating. One issue we discovered early with a client was that after installing Prerender, our crawlers were still running into a spider trap:

As you can probably tell, there were not actually 150,000 pages on that particular site. Our crawlers just found a recursive loop that kept generating longer and longer URL strings for the site content. This is something we would not have found in Google Search Console or Analytics. SPAs are notorious for causing tedious, inexplicable issues that you’ll only uncover by crawling the site yourself. Even if you follow the steps above and take as many precautions as possible, I can still almost guarantee you will come across a unique issue that can only be diagnosed through a crawl.

If you’ve come across any of these unique issues, let me know in the comments! I’d love to hear what other issues people have encountered with SPAs.

Results

As I mentioned earlier in the article, the process outlined above has enabled us to not only get client sites indexed, but even to get those sites ranking on first page for various keywords. Here’s an example of the keyword progress we made for one client with an AngularJS site:

Also, the organic traffic growth for that client over the course of seven months:

All of this goes to show that although SEO for SPAs can be tedious, laborious, and troublesome, it is not impossible. Follow the steps above, and you can have SEO success with your single-page app website.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

[ccw-atrib-link]

Darryl, the man behind dotmailer’s Custom Technical Solutions team

Why did you decide to come to dotmailer?

I first got to know dotmailer when the company was just a bunch of young enthusiastic web developers called Ellipsis Media back in 1999. I was introduced by one of my suppliers and we decided to bring them on board to build a recruitment website for one of our clients. That client was Amnesty International and the job role was Secretary General. Not bad for a Croydon company whose biggest client before that was Scobles the plumber’s merchants. So, I was probably dotmailer’s first ever corporate client! After that, I used dotmailer at each company I worked for and then one day they approached a colleague and me and asked us if we wanted to work for them. That was 2013.  We grabbed the opportunity with both hands and haven’t looked back since.

Tell us a bit about your role

I’m the Global Head of Technical Solutions which actually gives me responsibility for 2 teams. First, Custom Technical Solutions (CTS), who build bespoke applications and tools for customers that allow them to integrate more closely with dotmailer and make life easier. Second, Technical Pre-sales, which spans our 3 territories (EMEA, US and APAC) and works with prospective and existing clients to figure out the best solution and fit within dotmailer.

What accomplishments are you most proud of from your dotmailer time so far?

I would say so far it has to be helping to turn the CTS team from just 2 people into a group of 7 highly skilled and dedicated men and women who have become an intrinsic and valued part of the dotmailer organization. Also I really enjoy being part of the Senior Technical Management team. Here we have the ability to influence the direction and structure of the platform on a daily basis.

Meet Darryl Clark – the cheese and peanut butter sandwich lover

Can you speak a bit about your background and that of your team? What experience and expertise is required to join this team?

My background is quite diverse from a stint in the Army, through design college, web development, business analysis to heading up my current teams. I would say the most valuable skill that I have is being highly analytical. I love nothing more than listening to a client’s requirements and digging deep to work out how we can answer these if not exceed them.

As a team, we love nothing more than brainstorming our ideas. Every member has a valid input and we listen. Everyone has the opportunity to influence what we do and our motto is “there is no such thing as a stupid question.”

To work in my teams you have to be analytical but open minded to the fact that other people may have a better answer than you. Embrace other people’s input and use it to give our clients the best possible solution. We are hugely detail conscious, but have to be acutely aware that we need to tailor what we say to our audience so being able to talk to anyone at any level is hugely valuable.

How much of the dotmailer platform is easily customizable and when does it cross over into something that requires your team’s expertise? How much time is spent on these custom solutions one-time or ongoing?

I’ll let you in on a little secret here. We don’t actually do anything that our customers can’t do with dotmailer given the right knowledge and resources. This is because we build all of our solutions using the dotmailer public API. The API has hundreds of methods in both SOAP and REST versions, which allows you to do a huge amount with the dotmailer platform. We do have a vast amount of experience and knowledge in the team so we may well be able to build a solution quicker than our customers. We are more than happy to help them and their development teams build a solution using us on a consultancy basis to lessen the steepness of the learning curve.

Our aim when building a solution for a customer is that it runs silently in the background and does what it should without any fuss.

What are your plans for the Custom Tech Solutions team going forward?

The great thing about Custom Technical Solutions is you never know what is around the corner as our customers have very diverse needs. What we are concentrating on at the moment is refining our processes to ensure that they are as streamlined as possible and allow us to give as much information to the customer as we can. We are also always looking at the technology and coding approaches that we use to make sure that we build the most innovative and robust solutions.

We are also looking at our external marketing and sharing our knowledge through blogs so keep an eye on the website for our insights.

What are the most common questions that you get when speaking to a prospective customer?

Most questions seem to revolve around reassurance such as “Have you done this before?”, “How safe is my data?”, “What about security?”, “Can you talk to my developers?”, “Do I need to do anything?”.  In most instances, we are the ones asking the questions as we need to find out information as soon as possible so that we can analyse it to ensure that we have the right detail to provide the right solution.

Can you tell us about the dotmailer differentiators you highlight when speaking to prospective customers that seem to really resonate?

We talk a lot about working with best of breed so for example a customer can use our Channel Extensions in automation programs to fire out an SMS to a contact using their existing provider. We don’t force customers down one route, we like to let them decide for themselves.

Also, I really like to emphasize the fact that there is always more than one way to do something within the dotmailer platform. This means we can usually find a way to do something that works for a client within the platform. If not, then we call in CTS to work out if there is a way that we can build something that will — whether this is automating uploads for a small client or mass sending from thousands of child accounts for an enterprise level one.

What do you see as the future of marketing automation technology?  Will one size ever fit all? Or more customization going forward?

The 64 million dollar question. One size will never fit all. Companies and their systems are too organic for that. There isn’t one car that suits every driver or one racquet that suits every sport. Working with a top drawer partner network and building our system to be as open as possible from an integration perspective means that our customers can make dotmailer mold to their business and not the other way round…and adding to that the fact that we are building lots of features in the platform that will blow your socks off.

Tell us a bit about yourself – favorite sports team, favorite food, guilty pleasure, favorite band, favorite vacation spot?

I’m a dyed in the wool Gooner (aka Arsenal Football Club fan) thanks to my Grandfather leading me down the right path as a child. If you are still reading this after that bombshell, then food-wise I pretty much like everything apart from coriander which as far as I’m concerned is the Devils own spawn. I don’t really have a favorite band, but am partial to a bit of Level 42 and Kings of Leon and you will also find me listening to 90s drum and bass and proper old school hip hop. My favorite holiday destination is any decent villa that I can relax in and spend time with my family and I went to Paris recently and loved that. Guilty pleasure – well that probably has to be confessing to liking Coldplay or the fact that my favorite sandwich is peanut butter, cheese and salad cream. Go on try it, you’ll love it.

Want to meet more of the dotmailer team? Say hi to Darren Hockley, Global Head of Support, and Dan Morris, EVP for North America.

[ccw-atrib-link]

Becoming Better SEO Scientists – Whiteboard Friday

Posted by MarkTraphagen

Editor’s note: Today we’re featuring back-to-back episodes of Whiteboard Friday from our friends at Stone Temple Consulting. Make sure to also check out the second episode, “UX, Content Quality, and SEO” from Eric Enge.

Like many other areas of marketing, SEO incorporates elements of science. It becomes problematic for everyone, though, when theories that haven’t been the subject of real scientific rigor are passed off as proven facts. In today’s Whiteboard Friday, Stone Temple Consulting’s Mark Traphagen is here to teach us a thing or two about the scientific method and how it can be applied to our day-to-day work.

For reference, here’s a still of this week’s whiteboard.
Click on it to open a high resolution image in a new tab!

Video transcription

Howdy, Mozzers. Mark Traphagen from Stone Temple Consulting here today to share with you how to become a better SEO scientist. We know that SEO is a science in a lot of ways, and everything I’m going to say today applies not only to SEO, but testing things like your AdWords, how does that work, quality scores. There’s a lot of different applications you can make in marketing, but we’ll focus on the SEO world because that’s where we do a lot of testing. What I want to talk to you about today is how that really is a science and how we need to bring better science in it to get better results.

The reason is in astrophysics, things like that we know there’s something that they’re talking about these days called dark matter, and dark matter is something that we know it’s there. It’s pretty much accepted that it’s there. We can’t see it. We can’t measure it directly. We don’t even know what it is. We can’t even imagine what it is yet, and yet we know it’s there because we see its effect on things like gravity and mass. Its effects are everywhere. And that’s a lot like search engines, isn’t it? It’s like Google or Bing. We see the effects, but we don’t see inside the machine. We don’t know exactly what’s happening in there.

An artist’s depiction of how search engines work.

So what do we do? We do experiments. We do tests to try to figure that out, to see the effects, and from the effects outside we can make better guesses about what’s going on inside and do a better job of giving those search engines what they need to connect us with our customers and prospects. That’s the goal in the end.

Now, the problem is there’s a lot of testing going on out there, a lot of experiments that maybe aren’t being run very well. They’re not being run according to scientific principles that have been proven over centuries to get the best possible results.

Basic data science in 10 steps

So today I want to give you just very quickly 10 basic things that a real scientist goes through on their way to trying to give you better data. Let’s see what we can do with those in our SEO testing in the future.

So let’s start with number one. You’ve got to start with a hypothesis. Your hypothesis is the question that you want to solve. You always start with that, a good question in mind, and it’s got to be relatively narrow. You’ve got to narrow it down to something very specific. Something like how does time on page effect rankings, that’s pretty narrow. That’s very specific. That’s a good question. Might be able to test that. But something like how do social signals effect rankings, that’s too broad. You’ve got to narrow it down. Get it down to one simple question.

Then you choose a variable that you’re going to test. Out of all the things that you could do, that you could play with or you could tweak, you should choose one thing or at least a very few things that you’re going to tweak and say, “When we tweak this, when we change this, when we do this one thing, what happens? Does it change anything out there in the world that we are looking at?” That’s the variable.

The next step is to set a sample group. Where are you going to gather the data from? Where is it going to come from? That’s the world that you’re working in here. Out of all the possible data that’s out there, where are you going to gather your data and how much? That’s the small circle within the big circle. Now even though it’s smaller, you’re probably not going to get all the data in the world. You’re not going to scrape every search ranking that’s possible or visit every URL.

You’ve got to ask yourself, “Is it large enough that we’re at least going to get some validity?” If I wanted to find out what is the typical person in Seattle and I might walk through just one part of the Moz offices here, I’d get some kind of view. But is that a typical, average person from Seattle? I’ve been around here at Moz. Probably not. But this was large enough.

Also, it should be randomized as much as possible. Again, going back to that example, if I just stayed here within the walls of Moz and do research about Mozzers, I’d learn a lot about what Mozzers do, what Mozzers think, how they behave. But that may or may not be applicable to the larger world outside, so you randomized.

We want to control. So we’ve got our sample group. If possible, it’s always good to have another sample group that you don’t do anything to. You do not manipulate the variable in that group. Now, why do you have that? You have that so that you can say, to some extent, if we saw a change when we manipulated our variable and we did not see it in the control group, the same thing didn’t happen, more likely it’s not just part of the natural things that happen in the world or in the search engine.

If possible, even better you want to make that what scientists call double blind, which means that even you the experimenter don’t know who that control group is out of all the SERPs that you’re looking at or whatever it is. As careful as you might be and honest as you might be, you can end up manipulating the results if you know who is who within the test group? It’s not going to apply to every test that we do in SEO, but a good thing to have in mind as you work on that.

Next, very quickly, duration. How long does it have to be? Is there sufficient time? If you’re just testing like if I share a URL to Google +, how quickly does it get indexed in the SERPs, you might only need a day on that because typically it takes less than a day in that case. But if you’re looking at seasonality effects, you might need to go over several years to get a good test on that.

Let’s move to the second group here. The sixth thing keep a clean lab. Now what that means is try as much as possible to keep anything that might be dirtying your results, any kind of variables creeping in that you didn’t want to have in the test. Hard to do, especially in what we’re testing, but do the best you can to keep out the dirt.

Manipulate only one variable. Out of all the things that you could tweak or change choose one thing or a very small set of things. That will give more accuracy to your test. The more variables that you change, the more other effects and inner effects that are going to happen that you may not be accounting for and are going to muddy your results.

Make sure you have statistical validity when you go to analyze those results. Now that’s beyond the scope of this little talk, but you can read up on that. Or even better, if you are able to, hire somebody or work with somebody who is a trained data scientist or has training in statistics so they can look at your evaluation and say the correlations or whatever you’re seeing, “Does it have a statistical significance?” Very important.

Transparency. As much as possible, share with the world your data set, your full results, your methodology. What did you do? How did you set up the study? That’s going to be important to our last step here, which is replication and falsification, one of the most important parts of any scientific process.

So what you want to invite is, hey we did this study. We did this test. Here’s what we found. Here’s how we did it. Here’s the data. If other people ask the same question again and run the same kind of test, do they get the same results? Somebody runs it again, do they get the same results? Even better, if you have some people out there who say, “I don’t think you’re right about that because I think you missed this, and I’m going to throw this in and see what happens,” aha they falsify. That might make you feel like you failed, but it’s success because in the end what are we after? We’re after the truth about what really works.

Think about your next test, your next experiment that you do. How can you apply these 10 principles to do better testing, get better results, and have better marketing? Thanks.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

[ccw-atrib-link]

How to Use Server Log Analysis for Technical SEO

Posted by SamuelScott

It’s ten o’clock. Do you know where your logs are?

I’m introducing this guide with a pun on a common public-service announcement that has run on late-night TV news broadcasts in the United States because log analysis is something that is extremely newsworthy and important.

If your technical and on-page SEO is poor, then nothing else that you do will matter. Technical SEO is the key to helping search engines to crawl, parse, and index websites, and thereby rank them appropriately long before any marketing work begins.

The important thing to remember: Your log files contain the only data that is 100% accurate in terms of how search engines are crawling your website. By helping Google to do its job, you will set the stage for your future SEO work and make your job easier. Log analysis is one facet of technical SEO, and correcting the problems found in your logs will help to lead to higher rankings, more traffic, and more conversions and sales.

Here are just a few reasons why:

  • Too many response code errors may cause Google to reduce its crawling of your website and perhaps even your rankings.
  • You want to make sure that search engines are crawling everything, new and old, that you want to appear and rank in the SERPs (and nothing else).
  • It’s crucial to ensure that all URL redirections will pass along any incoming “link juice.”

However, log analysis is something that is unfortunately discussed all too rarely in SEO circles. So, here, I wanted to give the Moz community an introductory guide to log analytics that I hope will help. If you have any questions, feel free to ask in the comments!

What is a log file?

Computer servers, operating systems, network devices, and computer applications automatically generate something called a log entry whenever they perform an action. In a SEO and digital marketing context, one type of action is whenever a page is requested by a visiting bot or human.

Server log entries are specifically programmed to be output in the Common Log Format of the W3C consortium. Here is one example from Wikipedia with my accompanying explanations:

127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
  • 127.0.0.1 — The remote hostname. An IP address is shown, like in this example, whenever the DNS hostname is not available or DNSLookup is turned off.
  • user-identifier — The remote logname / RFC 1413 identity of the user. (It’s not that important.)
  • frank — The user ID of the person requesting the page. Based on what I see in my Moz profile, Moz’s log entries would probably show either “SamuelScott” or “392388” whenever I visit a page after having logged in.
  • [10/Oct/2000:13:55:36 -0700] — The date, time, and timezone of the action in question in strftime format.
  • GET /apache_pb.gif HTTP/1.0 — “GET” is one of the two commands (the other is “POST”) that can be performed. “GET” fetches a URL while “POST” is submitting something (such as a forum comment). The second part is the URL that is being accessed, and the last part is the version of HTTP that is being accessed.
  • 200 — The status code of the document that was returned.
  • 2326 — The size, in bytes, of the document that was returned.

Note: A hyphen is shown in a field when that information is unavailable.

Every single time that you — or the Googlebot — visit a page on a website, a line with this information is output, recorded, and stored by the server.

Log entries are generated continuously and anywhere from several to thousands can be created every second — depending on the level of a given server, network, or application’s activity. A collection of log entries is called a log file (or often in slang, “the log” or “the logs”), and it is displayed with the most-recent log entry at the bottom. Individual log files often contain a calendar day’s worth of log entries.

Accessing your log files

Different types of servers store and manage their log files differently. Here are the general guides to finding and managing log data on three of the most-popular types of servers:

What is log analysis?

Log analysis (or log analytics) is the process of going through log files to learn something from the data. Some common reasons include:

  • Development and quality assurance (QA) — Creating a program or application and checking for problematic bugs to make sure that it functions properly
  • Network troubleshooting — Responding to and fixing system errors in a network
  • Customer service — Determining what happened when a customer had a problem with a technical product
  • Security issues — Investigating incidents of hacking and other intrusions
  • Compliance matters — Gathering information in response to corporate or government policies
  • Technical SEO — This is my favorite! More on that in a bit.

Log analysis is rarely performed regularly. Usually, people go into log files only in response to something — a bug, a hack, a subpoena, an error, or a malfunction. It’s not something that anyone wants to do on an ongoing basis.

Why? This is a screenshot of ours of just a very small part of an original (unstructured) log file:

Ouch. If a website gets 10,000 visitors who each go to ten pages per day, then the server will create a log file every day that will consist of 100,000 log entries. No one has the time to go through all of that manually.

How to do log analysis

There are three general ways to make log analysis easier in SEO or any other context:

  • Do-it-yourself in Excel
  • Proprietary software such as Splunk or Sumo-logic
  • The ELK Stack open-source software

Tim Resnik’s Moz essay from a few years ago walks you through the process of exporting a batch of log files into Excel. This is a (relatively) quick and easy way to do simple log analysis, but the downside is that one will see only a snapshot in time and not any overall trends. To obtain the best data, it’s crucial to use either proprietary tools or the ELK Stack.

Splunk and Sumo-Logic are proprietary log analysis tools that are primarily used by enterprise companies. The ELK Stack is a free and open-source batch of three platforms (Elasticsearch, Logstash, and Kibana) that is owned by Elastic and used more often by smaller businesses. (Disclosure: We at Logz.io use the ELK Stack to monitor our own internal systems as well as for the basis of our own log management software.)

For those who are interested in using this process to do technical SEO analysis, monitor system or application performance, or for any other reason, our CEO, Tomer Levy, has written a guide to deploying the ELK Stack.

Technical SEO insights in log data

However you choose to access and understand your log data, there are many important technical SEO issues to address as needed. I’ve included screenshots of our technical SEO dashboard with our own website’s data to demonstrate what to examine in your logs.

Bot crawl volume

It’s important to know the number of requests made by Baidu, BingBot, GoogleBot, Yahoo, Yandex, and others over a given period time. If, for example, you want to get found in search in Russia but Yandex is not crawling your website, that is a problem. (You’d want to consult Yandex Webmaster and see this article on Search Engine Land.)

Response code errors

Moz has a great primer on the meanings of the different status codes. I have an alert system setup that tells me about 4XX and 5XX errors immediately because those are very significant.

Temporary redirects

Temporary 302 redirects do not pass along the “link juice” of external links from the old URL to the new one. Almost all of the time, they should be changed to permanent 301 redirects.

Crawl budget waste

Google assigns a crawl budget to each website based on numerous factors. If your crawl budget is, say, 100 pages per day (or the equivalent amount of data), then you want to be sure that all 100 are things that you want to appear in the SERPs. No matter what you write in your robots.txt file and meta-robots tags, you might still be wasting your crawl budget on advertising landing pages, internal scripts, and more. The logs will tell you — I’ve outlined two script-based examples in red above.

If you hit your crawl limit but still have new content that should be indexed to appear in search results, Google may abandon your site before finding it.

Duplicate URL crawling

The addition of URL parameters — typically used in tracking for marketing purposes — often results in search engines wasting crawl budgets by crawling different URLs with the same content. To learn how to address this issue, I recommend reading the resources on Google and Search Engine Land here, here, here, and here.

Crawl priority

Google might be ignoring (and not crawling or indexing) a crucial page or section of your website. The logs will reveal what URLs and/or directories are getting the most and least attention. If, for example, you have published an e-book that attempts to rank for targeted search queries but it sits in a directory that Google only visits once every six months, then you won’t get any organic search traffic from the e-book for up to six months.

If a part of your website is not being crawled very often — and it is updated often enough that it should be — then you might need to check your internal-linking structure and the crawl-priority settings in your XML sitemap.

Last crawl date

Have you uploaded something that you hope will be indexed quickly? The log files will tell you when Google has crawled it.

Crawl budget

One thing I personally like to check and see is Googlebot’s real-time activity on our site because the crawl budget that the search engine assigns to a website is a rough indicator — a very rough one — of how much it “likes” your site. Google ideally does not want to waste valuable crawling time on a bad website. Here, I had seen that Googlebot had made 154 requests of our new startup’s website over the prior twenty-four hours. Hopefully, that number will go up!

As I hope you can see, log analysis is critically important in technical SEO. It’s eleven o’clock — do you know where your logs are now?

Additional resources

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

[ccw-atrib-link]

MountainWest JavaScript 2014 – JavaScript SEO (The “Right” Way) by Jeff Whelpley

By Jeff Whelpley In the future, search engines may be smart enough to understand how to index JavaScript single page applications (SPAs). That is not the case today, however. Solutions based…

[ccw-atrib-link]

Everything You Need to Know About Mobile App Search

Posted by Justin_Briggs

Mobile isn’t the future. It’s the present. Mobile apps are not only changing how we interact with devices and websites, they’re changing the way we search. Companies are creating meaningful experiences on mobile-friendly websites and apps, which in turn create new opportunities to get in front of users.

I’d like to explore the growth of mobile app search and its current opportunities to gain visibility and drive engagement.

Rise of mobile app search

The growth of mobile device usage has driven a significant lift in app-related searches. This is giving rise to mobile app search as a vertical within traditional universal search.

While it has been clear for some time that mobile search is important, that importance has been more heavily emphasized by Google recently, as they continue to push
mobile-friendly labels in SERPs, and are likely increasing mobile-friendliness’s weight as a ranking factor.

The future of search marketing involves mobile, and it will not be limited to optimizing HTML webpages, creating responsive designs, and optimizing UX. Mobile SEO is a world where apps, knowledge graph, and conversational search are front and center.

For the
top 10 leading properties online, 34% of visitors are mobile-only (comScore data), and, anecdotally, we’re seeing similar numbers with our clients, if not more.

Mobile device and app growth

It’s also worth noting that
72% of mobile engagement relies on apps vs. on browsers. Looking at teen usage, apps are increasingly dominant. Additionally,
55% of teens use voice search more than once per day

If you haven’t read it, grab some coffee and read
A Teenagers View on Social Media, which is written by a 19-year old who gives his perspective of online behavior. Reading between the lines shows a number of subtle shifts in behavior. I noticed that every time I expected him say website, he said application. In fact, he referenced application 15 times, and it is the primary way he describes social networks.

This means that one of the fasting growing segments of mobile users cannot be marketed to by optimizing HTML webpages alone, requiring search marketers to expand their skills into app optimization.

The mobile app pack

This shift is giving rise to the mobile app pack and app search results, which are triggered on searches from mobile devices in instances of high mobile app intent. Think of these as being similar to local search results. Considering
mobile searcher behavior, these listings dominate user attention.

Mobile app search results and mobile app pack

As with local search, mobile app search can reorder traditional results, completely push them down, or integrate app listings with traditional web results.

You can test on your desktop using a
user-agent switcher, or by searching on your iOS or Android device. 

There are slight differences between iPhone and Android mobile app results:

iOS and Android mobile search result listing

From what I’ve seen, mobile app listings trigger more frequently, and with more results, on Android search results when compared to iOS. Additionally, iOS mobile app listings are represented as a traditional website result listing, while mobile app listings on Android are more integrated.

Some of the differences also come from the differences in app submission guidelines on the two major stores, the Apple App Store and Google Play.

Overview of differences in mobile app results

  1. Title – Google uses the app listing page’s HTML title (which is the app’s title). iOS app titles can exceed 55-62 characters, which causes wrapping and title truncation like a traditional result. Android app title requirements are shorter, so titles are typically shorter on Android mobile app listings.
  2. URL – iOS mobile app listings display the iTunes URL to the App Store as part of the search result.
  3. Icon – iOS icons are square and Android icons have rounded corners.
  4. Design – Android results stand out more, with an “Apps” headline above the pack and a link to Google Play at the end.
  5. App store content – The other differences show up in the copy, ratings, and reviews on each app store.

Ranking in mobile app search results

Ranking in mobile app search results is a
combination of App Store Optimization (ASO) and traditional SEO. The on-page factors are dependent upon your app listing, so optimization starts with having solid ASO. If you’re not familiar with ASO, it’s the process of optimizing your app listing for internal app store search.

Basics of ASO

Ranking in the Apple App Store and in Google Play is driven by two primary factors: keyword alignment and app performance. Text fields in the app store listing, such as title, description, and keyword list, align the app with a particular set of keywords. Performance metrics including download velocity, app ratings, and reviews determine how well the app will rank for each of those keywords. (Additionally, the Google Play algorithm may include external, web-based performance metrics like citations and links as ranking factors.)

App store ranking factors

Mobile app listing optimization

While I won’t explore ASO in-depth here, as it’s very similar to traditional SEO,
optimizing app listings is primarily a function of keyword targeting.

Tools like
Sensor Tower, MobileDevHQ, and App Annie can help you with mobile app keyword research. However, keep in mind that mobile app search listings show up in universal search, so it’s important to leverage traditional keyword research tools like the AdWords Tool or Google Trends.

While there are similarities with ASO, optimizing for these mobile app search listings on the web has some slight differences.

Differences between ASO & mobile app SEO targeting

  1. Titles – While the Apple App Store allows relatively long titles, they are limited to the preview length in organic search. Titles should be optimized with Google search in mind, in addition to optimizing for the app store. Additionally, several apps aggressively target keywords in their app title, but caution should be used as spamming keywords could influence app performance in Google.
  2. Description – The app description on the App Store may not be a factor in internal search, but it will impact external app search results. Leverage keyword targeting best practices when writing your iOS app description, as well as your Android app description.
  3. Device and platform keywords – When targeting for app store search, it is not as important to target terms related to the OS or device. However, these terms can help visibility in external search. Include device and OS terms, such as Android, Samsung Note, iOS, iPad, and iPhone.

App performance optimization

Outside of content optimization, Google looks at the performance of the app. On the Android side, they have access to the data, but for iOS they have to rely on publicly available information.

App performance factors

  • Number of ratings
  • Average rating score
  • Content and sentiment analysis of reviews
  • Downloads / installs
  • Engagement and retention
  • Internal links on app store

For iOS, the primary public metrics are ratings and reviews. However, app performance can be inferred using the App Store’s ranking charts and search results, which can be leveraged as proxies of these performance metrics.


The following objectives will have the greatest influence on your mobile app search ranking:

  1. Increase your average rating number
  2. Increase your number of ratings
  3. Increase downloads

For app ratings and reviews, leverage platforms like
Apptentive to improve your ratings. They are very effective at driving positive ratings. Additionally, paid tactics are a great way to drive install volume and are one area where paid budget capacity could directly influence organic results in Google. Anecdotally, both app stores use rating numbers (typically above or below 4 stars) to make decisions around promoting an app, either through merchandising spots or co-branded campaigns. I suspect this is being used as a general cut-off for what is displayed in universal results. Increasing your rating above 4 stars should improve the likelihood you’ll appear in mobile app search results.

Lastly, think of merchandising and rankings in terms of 
internal linking structures. The more visible you are inside of the app store, the more visibility you have in external search.

App web performance optimization

Lastly, we’re talking Google rankings, so factors like links, citations, and social shares matter. You should be
conducting content marketing, PR, and outreach for your app. Focus on merchandising your app on your own site, as well as increasing coverage of your app (linking to the app store page). The basics of link optimization apply here.

App indexation – drive app engagement

Application search is not limited to driving installs via app search results. With app indexing, you can leverage your desktop/mobile website visibility in organic search to drive engagement with those who have your app installed. Google can discover and expose content deep inside your app directly in search results. This means that when a user clicks on your website in organic search, it can open your app directly, taking them to that exact piece of content in your app, instead of opening your website.

App indexation fundamentally changes technical SEO, extending SEO from server and webpage setup to the setup and optimization of applications.

App indexation on Google

This also fundamentally changes search. Your most avid and engaged user may choose to no longer visit your website. For example, on my Note 4, when I click a link to a site of a brand that I have an app installed for, Google gives me the option not only to open in the app, but to set opening the app as a default behavior.

If a user chooses to open your site in your app, they may never visit your site from organic search again.

App indexation is currently limited to Android devices, but there is evidence to suggest that it’s already in the works and is
soon to be released on iOS devices. There have been hints for some time, but markup is showing up in the wild suggesting that Google is actively working with Apple and select brands to develop iOS app indexing.

URI optimization for apps

The first step in creating an indexable app is to set up your app to support deep links. Deep links are URIs that are understood by your app and will open up a specific piece of content. They are effectively URLs for applications.

Once this URI is supported, a user can be sent to deep content in the app. These can be discovered as alternates to your desktop site’s URLs, similar to how
separate-site mobile sites are defined as alternate URLs for the desktop site. In instances of proper context (on an Android device with the app installed), Google can direct a user to the app instead of the website.

Setting this up requires working with your app developer to implement changes inside the app as well as working with your website developers to add references on your desktop site.

Adding intent filters

Android has
documented the technical setup of deep links in detail, but it starts with setting up intent filters in an app’s Android manifest file. This is done with the following code.

<activity android:name="com.example.android.GizmosActivity"
android:label="@string/title_gizmos" >
<intent-filter android:label="@string/filter_title_viewgizmos">
<action android:name="android.intent.action.VIEW" />
<data android:scheme="http"
android:host="example.com"
android:pathPrefix="/gizmos" />
<category android:name="android.intent.category.DEFAULT" />
<category android:name="android.intent.category.BROWSABLE" />
</intent-filter>
</activity>

This dictates the technical optimization of your app URIs for app indexation and defines the elements used in the URI example above.

  • The <intent-filter> element should be added for activities that should be launchable from search results.
  • The <action> element specifies the ACTION_VIEW intent action so that the intent filter can be reached from Google Search.
  • The <data> tag represents a URI format that resolves to the activity. At minimum, the <data> tag must include the android:scheme attribute.
  • Include the BROWSABLE category. The BROWSABLE category is required in order for the intent filter to be accessible from a web browser. Without it, clicking a link in a browser cannot resolve to your app. The DEFAULT category is optional, but recommended. Without this category, the activity can be started only with an explicit intent, using your app component name.

Testing deep links

Google has created tools to help test your deep link setup. You can use
Google’s Deep Link Test Tool to test your app behavior with deep links on your phone. Additionally, you can create an HTML page with an intent:// link in it.

For example
:

<a href="intent://example.com/page-1#Intent;scheme=http;package=com.example.android;end;"> <a href="http://example.com/page-1">http://example.com/page-1></a>

This link would open up deep content inside the app from the HTML page.

App URI crawl and discovery

Once an app has deep link functionality, the next step is to
ensure that Google can discover these URIs as part of its traditional desktop crawling.

Ways to get apps crawled

  1. Rel=”alternate” in HTML head
  2. ViewAction with Schema.org
  3. Rel=”alternate” in XML Sitemap

Implementing all three will create clear signals, but at minimum you should add the rel=”alternate” tag to the HTML head of your webpages.

Effectively, think of the app URI as being similar to a mobile site URL when
setting up a separate-site mobile site for SEO. The mobile deep link is an alternative way to view a webpage on your site. You map a piece of content on your site to a corresponding piece of content inside the app.

Before you get started, be sure to
verify your website and app following the guidelines here. This will verify your app in Google Play Developer Console and Google Webmaster Tools.

#1: Rel=”alternate” in HTML head

On an example page, such as example.com/page-1, you would add the following code to the head of the document. Again, very similar to separate-site mobile optimization.

<html>
<head> 
... 
<link rel="alternate" href="android-app://com.example.android/http/example.com/page-1" /> 
...
</head>
<body>
</body>
#2: ViewAction with Schema.org

Additionally, you can reference the deep link using Schema.org and JSON by using a 
ViewAction.

<script type="application/ld+json"> 
{ 
"@context": "http://schema.org", 
"@type": "WebPage", 
"@id": "http://example.com/gizmos", 
"potentialAction": { 
"@type": "ViewAction", 
"target": "android-app://com.example.android/http/example.com/gizmos" 
} 
} 
</script>
#3 Rel=”alternate” in XML sitemap

Lastly, you can reference the alternate URL in your XML Sitemaps, similar to using the rel=”alternate” for mobile sites.

<?xml version="1.0" encoding="UTF-8" ?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"> 
<url> 
<loc>http://example.com/page-1</loc> 
<xhtml:link rel="alternate" href="android-app://com.example.android/http/example.com/page-1" /> 
</url> 
... 
</urlset>

Once these are in place, Google can discover the app URI and provide your app as an alternative way to view content found in search.

Bot control and robots noindex for apps

There may be instances where there is content within your app that you do not want indexed in Google. A good example of this might be content or functionality that is built out on your site, but has not yet been developed in your app. This would create an inferior experience for users. The good news is that we can block indexation with a few updates to the app.

First, add the following to your app resource directory (res/xml/noindex.xml).

<?xml version="1.0" encoding="utf-8"?> 
<search-engine xmlns:android="http://schemas.android.com/apk/res/android"> 
<noindex uri="http://example.com/gizmos/hidden_uri"/> 
<noindex uriPrefix="http://example.com/gizmos/hidden_prefix"/> 
<noindex uri="gizmos://hidden_path"/> 
<noindex uriPrefix="gizmos://hidden_prefix"/> 
</search-engine>

As you can see above, you can block an individual URI or define a URI prefix to block entire folders.

Once this has been added, you need to update the AndroidManifest.xml file to denote that you’re using noindex.html to block indexation.

<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example.android.Gizmos"> 
<application> 
<activity android:name="com.example.android.GizmosActivity" android:label="@string/title_gizmos" > 
<intent-filter android:label="@string/filter_title_viewgizmos"> 
<action android:name="android.intent.action.VIEW"/> 
... 
</activity> 
<meta-data android:name="search-engine" android:resource="@xml/noindex"/> 
</application> 
<uses-permission android:name="android.permission.INTERNET"/> 
</manifest>

App indexing API to drive re-engagement

In addition to URI discovery via desktop crawl, your mobile app can integrate
Google’s App Indexing API, which communicates with Google when users take actions inside your app. This sends information to Google about what users are viewing in the app. This is an additional method for deep link discovery and has some benefits.

The primary benefit is the ability to appear in
autocomplete. This can drive re-engagement through Google Search query autocompletions, providing access to inner pages in apps.

App auto suggest

Again, be sure to
verify your website and app following the guidelines here. This will verify your app in Google Play Developer Console and Google Webmaster Tools.

App actions with knowledge graph

The next, and most exciting, evolution of search is leveraging actions. These will be powerful when
combined with voice search, allowing search engines to take action on behalf of users, turning spoken language into executed actions.

App indexing allows you to take advantage of actions by allowing Google to not only launch an app, but execute actions inside of the app. Order me a pizza? Schedule my meeting? Drive my car? Ok, Google.

App actions work via entity detection and the application of the knowledge graph, allowing search engines to understand actions, words, ideas and objects. With that understanding, they can build an action graph that allows them to define common actions by entity type.

Here is a list of actions currently supported by Schema.org

For example, the PlayAction could be used to play a song in a music app. This can be achieve with the following markup.

<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "MusicGroup",
"name": "Weezer", "potentialAction": {
"@type": "ListenAction",
"target": "android-app://com.spotify.music/http/we.../listen"
}
}
</script>
Once this is implemented, these app actions can begin to appear in search results and knowledge graph.

deep links in app search results

Overview of mobile app search opportunities

In summary, there are five primary ways to increase visibility and engagement for your mobile app in traditional organic search efforts.

Mobile apps in search results

The growth of mobile search is transforming how we define technical SEO, moving beyond front-end and back-end optimization of websites into the realm of structured data and application development. As app indexing expands to include iOS, I suspect the possibilities and opportunities associated with indexing applications, and their corresponding actions, to grow extensively. 

For those with Android apps, app indexing is a potential leapfrog style opportunity to get ahead of competitors who are dominant in traditional desktop search. Those with iOS devices should start by optimizing their app listings, while preparing to implement indexation, as I suspect it’ll be released for iOS this year.

Have you been leveraging traditional organic search to drive visibility and engagement for apps? Share your experiences in the comments below.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

[ccw-atrib-link]