Posts

robots txt file

The Importance of a Robots TXT File for your Website 


Increasing traffic to your website and staying on top of SEO is important for your business, but we know just how much of a rigorous process it can be. It takes a lot of analysis and patience, and between all the Google tools and intricate coding, the technicality of it all can get confusing and even overwhelming sometimes. To stay on top, the job is never done!

But what if we told you there is something that could make it a little bit easier? It’s called a robots.txt file, and in a nutshell, it’s a useful tool that gives you control on how search engines navigate or “crawl” through your site, and allows them to do so more efficiently.

It might sound intimidating, but it’s actually not as complicated as it seems, and you don’t have to be super tech-savvy to take advantage. Here we will break it all down for you, and explain exactly what a robots txt file is and how you can use it. Get ready—it’s a game-changer!

 
What is a robots.txt file?

The robots.txt file is a text file that tells web robots, usually search engine robots like Googlebot, which pages to crawl through on a website, and which ones not to crawl. Also referred to as the robot exclusion protocol, it is part of a larger group of web standards that controls how robots crawl the web, access and interpret content, and display that content to users. In this case with the robots.txt file, this is done by “disallowing” or “allowing” the actions of certain or all robots and user agents (web crawling software). 

This is the basic body of the file:

User-agent: *
Disallow: /wp-admin/


The asterisk after “user-agent” means that the file applies to all web robots that visit the site, and the slash after “disallow” informs robots which pages they can’t, or aren’t allowed to follow—which, in this case means only the admin login page of this WordPress site (we know it’s a WordPress site because of the /wp-admin). 

This is a complete robots.txt file on its own, but you can get more specific and add more lines of directives if needed. We’ll touch more on this later. 

Consider the robots txt file to be a function that is meant to prevent your site from being overloaded with requests. It is not meant to stop your website from showing up on Google. There are other functions to keep your site out of Google, such as “no indexing” the site.

Why is the Robots File Important?

Along with keeping your website running smoothly, there are many other benefits to having control over site crawlers. Some of these could include:

  • Keeping parts of your website private (for example, an internal or admin login page, or a duplicate printer-friendly version of a page).
  • Restricting a page on your site that you don’t want users to access unless they take a specific action.
  • Linking to your XML sitemap(s).
  • Preventing certain files such as images or PDFs from getting indexed and showing up on search engines.
  • Implementing a crawl delay to prevent your servers from being overloaded when multiple pieces of content are being loaded at once.


And that’s just to name a few! It is also worth considering that if you don’t give directions with a robots.txt file, search engines will have to take more time to unnecessarily crawl through every single page on your site, which will slow them down and make your site seem less reliable. All of this could put you lower in the search results. So why waste time? If you make it easier for the search bots, you’ll be rewarded.
 

How to Find and Create a Robots.txt File

Now that we’ve piqued your interest, it’s time to get your robots.txt file started.

The first thing is to know how to access it. All that you have to do is enter the URL of your website into the address bar and add “/robots.txt” to the end—like this: “ideazone.ca/robots.txt.” Head to the page, and you’ll find that it is either empty (meaning you don’t have a file) or that you already have a robots.txt file.

If you don’t have one, not to worry; it’s easy to create one from scratch. You can use any plain text editor, such as Notepad, TextEdit, or Editpad.org, and enter the lines of code there. 


PRO TIP: Note that using a program like Microsoft Word is a big NO, as it could add additional code into the text that you don’t want.


If you do have a robots.txt file already, you can find it in your site’s root directory by going to your hosting account website, logging in, and heading to the file management section of your webpage.

Now, it’s time to begin with the basic format that we mentioned above. But you’re going to want to be more specific and only disallow certain pages. To do that, you would simply use the URL of the page, like this:

User-agent: *
Disallow: /wp-admin/

 

Notice that each command/directive is on a separate line.

With Googlebot, you can allow certain pages, as well:


User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-example.php


Many robots txt files look just like this—short and simple. But once you get familiar with the syntax and language, you can get more advanced and include many more directives if you wish. Adding your sitemap and a crawl-delay function could be a good idea. To learn more about the technical syntax, Google is always a great resource.

 

We have just scratched the surface here, but hopefully this has given you a thorough introduction and encouraged you to set up a robots.txt file for your website. Whether it’s a text file you need, website design, or to develop an effective digital marketing strategy, our team of experts at IdeaZone can help you build a high-ranking website that delivers results. Call or email us today to learn how we can give your business a boost.

amp mobile pages logo

AMP Has a Large List of Pros and Cons

What is AMP?

AMP amplifies the mobile user experience (UX) by bringing your content to your users instantly on mobile devices by shrinking and caching your website. This speeds up the loading process for sites, ads and emails to less than 1 second. This instant gratification means the overall user experience is improved with less drain on battery life and data usage. If AMP is the right choice for your site, it can help reduce bounce rates and encourage users to complete a transaction.

Originally known as Accelerated Mobile Pages, the AMP Project backed by Google is now called by its acronym, AMP. It’s a hotly debated topic, with many points of contention. We’ll cover some of the main pros and cons in this article, beginning with the good stuff and why AMP was created in the first place.

AMP – The Good

Mobile Web Browsing Continues to Rise

In 2017, a report by Forrester Consulting showed that users spend 2X more time on an AMP page and AMP brings a 10% increase in web traffic.

Consider the last time you shopped for information or a product online. Did you use your phone or your laptop / desktop PC? According to Urban Airship, studies show that mobile internet usage has surpassed desktop mobile usage all over the globe.

Even in developing countries like Indonesia and Mexico, customers expect an immediate answer and use their smartphone for web searches 4x as much as desktop PCs. The reasoning is simple. Customers have their smartphones with them day and night because they are now the ultimate portable information gathering device. AMP can help ensure your customers:

  1. Visit your site
  2. Stay on your site for reduced bounce rates
  3. Complete their transaction
  4. Come back!

AMP Formatting is Simple

Using AMP can be your chance to offer your users what they want right from their mobile searches.  It works because AMP does away with all the clunky desktop-style elements that are still included with mobile-optimized pages today. This slimmed-down version gets rid of all the fluff and layers.

Restrictions on JavaScript, images and on other dynamic video content is the secret sauce that creates immediate viewing with faster load times of about 1 second. 

AMP is Compatible With Major Web Browsers and Platforms

Firefox, Safari, Edge, Opera, UC Browser and of course, Google Chrome. AMP also works with the biggest platforms, including:

  • LinkedIn
  • Medium
  • Reddit
  • Tumblr
  • Bing

There are 4 types of Google AMP formats to instantly share important news and visual narratives on mobile pages, including:

  • Ads
  • Emails
  • Stories and Blog Posts
  • Websites / Web Pages

Feature – Rich Content

Instead of the basic blue links you’d usually see in Google search results on a non-AMP page, you’ll see feature-rich content. One thing you might notice right away is the storytelling format. This content format is known as a carousel. Similar to image carousels seen in Facebook or Instagram ads, these AMP carousels serve to get your viewers clicking through the content, right onto your page. 

AMP – The Bad

AMP Isn’t a True Ranking Factor

AMP isn’t without its drawbacks. Even though site loading speed is a ranking factor for SEO,  AMP isn’t a true ranking factor.

Canonical AMP Pages Show as Standard Results

Another thing to keep in mind is that AMP pages only show as standard results on desktop so if your customers switch from mobile to a slower desktop connection, they won’t get the same AMP experience from your site.

AMP and Google Analytics are Complex Cousins

AMP and Google Analytics can be set up, but it’s not a simple process for the average website owner. Adding the proper tag can be difficult and time-consuming. If not done properly, you’ll easily miss out on important and relevant data. 

Signed Exchange is Limited

With Signed Exchange (SXG) the AMP page will be treated as if it belongs to your site. Without using signed exchange, your site will simply show under the new AMP page URL instead of your own URL. Signed exchange is also only supported in Google search for basic results and rich results, not carousels. It’s also only used with Chrome which makes it rather limiting.

AMP – The Nasty

AMP Hosts Everything Within Google’s Server

Because of this, you’re at the mercy of Google’s design. Your ads, stories etc. on AMP pages show the way Google chooses. This can mean your chosen design elements aren’t carried through. You’ll notice that your content is shown as somewhat boilerplate, following the basic AMP format. While AMP is more streamlined, it can really affect all your hard work in SEO and design, affecting how you want your customers to access and click through your site.

AMP Pages Look a Little Too Similar

Mobile news consumption is a big part of what mobile users want. Generally, it’s a good way to quickly see the top stories and snippets, with the option to swipe into the main website. However, because content looks so similar, it can also be easier for clickbait and fake news stories to appear alongside legitimate AMP results.

“All publishers end up looking more similar than different. That makes separating the real from the fake even harder,” says tech writer Kyle Chayka, who has written for the New Yorker, Harper’s and other highly reputable publications.

Domain Authority and Backlinks Go to Google

Because AMP hosts everything within Google’s server, any domain authority or backlinks go to Google and not your site. This is another strike against AMP framework for site owners and developers who work hard to create and maintain a stellar domain authority and SEO score. It also indicates that there’s a large loss of control over your own website if you exclusively use AMP and don’t have an active non-AMP version of your site.

AMP Tends to Fight With IOS

Users have said the scrolling patterns and behaviour on IOS feel unnatural and make it impossible to tap the status bar to instantly scroll to the top of the page and to be able to search for a term within the page. Many users have reported that AMP pages don’t render properly on their mobile devices and that content shows up as tiny columns of unreadable text. 

AMP Could Affect Ad Revenue

In an article by Digiday, a Google spokesperson didn’t deny that the number one goal of AMP is to improve user experience by loading content first and ads second. The problem with loading ads second is that ads have multiple tracking tags and higher-res images that take longer to load anyways. If the user has gone ahead and clicked on the main content, they haven’t even seen the ad. Also, the quicker the content loads, the more the ad loading speed will lag. Google has made steps towards a fix for this common issue, but some publishers may still experience slower ad loading times. 

The main thing to note here is that AMP is a very mixed bag.  These are only a few of the main sticking points, but by now you probably know whether it’s right for your site. 

If your visitors are mostly desktop users, hang tight, keep doing what you’re doing. We’d be glad to discuss how we can boost your traffic, SEO ratings and your website as a whole, for both desktop AND mobile users. If you’re unsure how fast your mobile site is loading, you can use this site speed test. 

If you have a lot of mobile users, like the idea of having an instant news feed (similar to Facebook Instant Articles and Apple News) and like the idea of a diet, bite-sized version of your website, we’re happy to help with that too. Together we can discuss your options for different types of AMP pages, eye-catching carousels, platforms, analytics gathering techniques and more as AMP technology continues to evolve.

Questions about AMP and building your website? At IdeaZone, we’re your secret weapon for getting the most out of your online presence. For more information and to get in touch with our team of Victoria, BC digital marketing experts, contact us via web form or toll free at 1-844-651-9333.