Thursday, 29 January 2015

Our Bots

Our bots - svachalit programas
A bot (short for "robot") is a program that operates as an agent for a user or another program or simulates a human activity.
On the Internet, the most ubiquitous bots are the programs, also called Spiders or Crawlers that access Web sites and gather their content for Search Engine indexes.
A bot is an automated software application which typically performs tasks over the Internet. There is virtually an unlimited number of bots performing a dizzying array of tasks. In the world of online marketing, we see bots used to crawl websites, scrape content, check search rankings, and automate social media and much more.
In some cases bots are good things offering automation which enables huge advances in access to information (think search engine bots). In other cases bots can be “evil” by stealing content, breaking websites and costing business owners a ton of money.
It is always best to have permission before using any bot on a website other than your own.


Twitterbot is a program used to produce automated posts on the Twitter micro blogging service, or to automatically follow Twitter users. Twitter bots come in various forms. For example, many serve as spam, enticing clicks on promotional links. Others post @replies or automatically "retweet" in response to tweets that include a certain word or phrase. These automatic tweets are often seen as fun or silly. Some Twitter users even program Twitter bots to assist themselves with scheduling or reminders.

Features of a Twitterbot

It is sometimes desirable to identify when a Twitter account is controlled by a bot. In a 2012 paper, Chu et al. propose the following criteria that indicate that an account may be a bot (they were designing an automated system):
·         "Periodic and regular timing" of tweets;
·         Whether the tweet content contains known spam; and
·         The ratio of tweets from mobile versus desktop, as compared to an average human Twitter user.
There are many different types of Twitter bots and their purposes vary from one to another. Some bots may tweet helpful material such as @EarthquakesSF (description below). 
·        @chatmundo is an AI conversational Twitter bot based on Program O which responds to @chatmundo mentions.
·        @WBEZbot Tweets the current programming on NPR’s Chicago affiliate station, WBEZ.
·        @KookyScrit sends auto-reply tweets correcting misspellings of the word "weird." @choose_this sends at-replies to twitter users who tweet about making a choice between wide varieties of things.
Twitter Bots
Automatically follow un-follow, retweet, and gather IDs and more.
The Good: Build a following fast and automatically generate content for your tweets (or retweets).
The Bad: If your followers realize they’ve followed a bot you’re not likely to make many friends. Excessive bot usage can get you banned from Twitter.
Examples: tweet adder, Injek Twit.
Website Scrapers
Identify and download specific strings of text or images from a website.
The Good: Save marketing admins countless hours of copy and paste work by scraping website content yourself.
The Bad: Scraping is normally associated with taking other people’s content to use as your own or scraping a website without permission or in violation of the website’s TOS or AUP.
Examples: Website Content Extractor,
Website Crawlers / Scrapers – Search Engines (indexing software)
Bots sent by search engines to browse and store content from your website. This content is then used to help rank your website on said search engine.
The Good: Search engines exist, and I’m guessing you get a good percentage of your traffic from search engines.
The Bad: You have to learn how this little bot works to make sure your site is accessible and easily navigated by the various bots. Think linking structure and indexability.
Examples: 80legs
Search Rank Checkers
Used to check the position of your organic and paid listings on search engines.
The Good: Learn how you’re ranking across a variety of search engines, countries, languages and more. When used appropriately with an API you will not have to worry about TOS/AUP violations.
The Bad: When not used appropriately, or without an API, search rank checkers may get your IP banned or worse, your website. Be nice to search engines and try to play by the rules.
Examples: Rank Tracker, Rank Reporter, Web Position Reporter
Facebook Bots
Mass friend requests, messaging, wall posting, poking, status updates and more.
The Good: Build your friend / fan lists quickly and automatically update your status, images and more.
The Bad: Everything else. Automated mass friend requesting is a good way to get your account banned and automated pokes just sounds painful.
Example: Facebook Blaster Pro 
Comment Spam Bots
Used to post comment spam on blogs, forums and news websites for the purpose of link building.
The Good: None.
The Bad: Your blog is taken over by comments like “I really like your article. You should check out my website about Cialis” Also, if you get caught using a comment spam spot Matt Cutts is likely to ban you from Google and flame you on his blog. Comment spams that blog post in retort.

PPC Bots
Click on your competitor’s ad, influence bounce rates, and generally cause havoc with Ad words accounts.
The Good: None
The Bad: Not only are people, who use PPC bots defrauding advertisers, they are also at great risk of being banned by search engines.
Link Building Bots
Find websites and automatically email webmasters requesting back links.
The Good: Build links automatically.
The Bad: You’ll end up annoying countless webmasters who may end up posting negative information about your website. Couple this with the fact that you’ll be a low rate spammer a hair’s breadth away from being banned by most search engines and link building bots are better left alone.
Bots offer a powerful way for you to automate repetitive tasks; however, think long and hard about using bots with third party websites or services. As mentioned before, check the terms of service, Acceptable Use Policy and any applicable laws. It is always best to have permission before using a bot on any website other than your own.

What You Should Know About Google Bots and SEO

Ad clicks aren’t the only pressing issue for websites when it comes to online marketing and bots – since tools like Google analytics don’t provide granular insight into site traffic, it can be very tricky to differentiate between human and non-human traffic unless you really drill down, or notice after the fact that something is amiss. Even more difficult is deciphering between malicious bots that can harm your website and good bots, like Googlebot, that improve SEO.
If you run an e-commerce site has website or a personal blog, chances are you want Google to visit your site and index your content as often as possible. By doing so, Google learns what is new on your site and can immediately share updated content with the online community. However, differentiating between Google and hackers who impersonate Google can be a major challenge for website operators and can have a damaging impact on your online business.
The Googlebot has a very unique way of identifying itself. It uses a specific user agent that arrives from IP addresses belonging to Google and always adheres to the robots.txt (the crawling instructions that website owner provide to such bots).
Google uses a robot called “Googlebot” that crawls millions of sites simultaneously and indexes their content in Google’s databases. The more Googlebot visits your site, the faster your site’s content updates will appear in Google’s search results. It’s crucial to allow Googlebot to crawl your website without blocking or disturbing it, and many companies invest in special SEO tools to attract it.

What is Robots.txt?

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

Cheat Sheet

Block all web crawlers from all content
User-agent: * 
Disallow: /
Block a specific web crawler from a specific folder
User-agent: Googlebot 
Disallow: /no-google/
Block a specific web crawler from a specific web page
User-agent: Googlebot 
Disallow: /no-google/blocked-page.html
Sitemap Parameter
User-agent: * 

Optimal Format

Robots.txt needs to be placed in the top-level directory of a web server in order to be useful. Example:

What is Robots.txt?

The Robots Exclusion Protocol (REP) is a group of web standards that regulate web robot behavior and search engine indexing. The REP consists of the following:
  • The original REP from 1994, extended 1997, defining crawler directives for robots.txt. Some search engines support extensions like URI patterns (wild cards).
  • Its extension from 1996 defining indexer directives (REP tags) for use in the robots Meta element, also known as "robots meta tag." Meanwhile, search engines support additional REP tags with an X-Robots-Tag. Webmasters can apply REP tags in the HTTP header of non-HTML resources like PDF documents or images.
  • The Microformat rel-nofollow from 2005 defining how search engines should handle links where the A Element's REL attributes contains the value "nofollow."

Robots Exclusion Protocol Tags

Applied to an URI, REP tags (noindex, nofollow, unavailable_after) steer particular tasks of indexers, and in some cases (nosnippet, noarchive, noodp) even query engines at runtime of a search query. Other than with crawler directives, each search engine interprets REP tags differently. For example, Google wipes out even URL-only listings and ODP references on their SERPs when a resource is tagged with "noindex," but Bing sometimes lists such external references to forbidden URLs on their SERPs. Since REP tags can be supplied in META elements of X/HTML contents as well as in HTTP headers of any web object, the consensus is that contents of X-Robots-Tags should overrule conflicting directives found in META elements.


Indexer directives put as microformats will overrule page settings for particular HTML elements. For example, when a page's X-Robots-Tag states "follow" (there's no "nofollow" value), the rel-nofollow directive of a particular an element (link) wins.

Although robots.txt lacks indexer directives, it is possible to set indexer directives for groups of URIs with server sided scripts acting on site level that apply X-Robots-Tags to requested resources. This method requires programming skills and good understanding of web servers and the HTTP protocol.

Pattern Matching

Google and Bing both honor two regular expressions that can be used to identify pages or sub-folders that an SEO wants excluded. These two characters are the asterisk (*) and the dollar sign ($).
  • * - which is a wildcard that represents any sequence of characters
  • $ - which matches the end of the URL

Public Information

The robots.txt file is public—be aware that a robots.txt file is a publicly available file. Anyone can see what sections of a server the webmaster has blocked the engines from. This means that if an SEO has private user information that they don’t want publicly searchable, they should use a more secure approach—such as password protection—to keep visitors from viewing any confidential pages they don't want indexed.

Important Rules

  • In most cases, meta robots with parameters "noindex, follow" should be employed as a way to to restrict crawling or indexation.
  • It is important to note that malicious crawlers are likely to completely ignore robots.txt and as such, this protocol does not make a good security mechanism.
  • Only one "Disallow:" line is allowed for each URL.
  • Each subdomain on a root domain uses separate robots.txt files.
  • Google and Bing accept two specific regular expression characters for pattern exclusion (* and $).
  • The filename of robots.txt is case sensitive. Use "robots.txt", not "Robots.TXT."
  • Spacing is not an accepted way to separate query parameters. For example, "/category/ /product page" would not be honored by robots.txt.

SEO Best Practice

Blocking Page

There are a few ways to block search engines from accessing a given domain:
Block with Robots.txt
This tells the engines not to crawl the given URL, but that they may keep the page in the index and display it in in results. (See image of Google results page below.)
Block with Meta NoIndex
This tells engines they can visit, but are not allowed to display the URL in results. This is the recommended method.
Block by Nofollowing Links
This is almost always a poor tactic. Using this method, it is still possible for the search engines to discover pages in other ways: through browser toolbars, links from other pages, analytics, and more.

Why Meta Robots is better than Robots.txt?

Below is an example of's robots.txt file. Notice that they are blocking the directory /library/nosearch/.

Now notice what happens when the URL is searched for in Google.
Google has 2,760 pages from that "disallowed" directory. The engine hasn't crawled these URLs, so it appears as a URL rather than a traditional listing.
This becomes a problem when these pages accumulate links. Those pages than can accumulate link juice (ranking power) and other query-independent ranking metrics (like popularity and trust), but these pages can't pass these benefits to any other pages since the links on them don't ever get crawled.
In order to exclude individual pages from search engine indices, the noindex meta tag <meta name="robots" content="noindex"> is actually superior to robots.txt.

Thursday, 22 January 2015

To get Response is the future

To get Response is the future

What is Responsive Web Design?
Responsive Web Design (RWD) is an approach of laying-out and coding a website such that the website provides an optimal viewing experience — ease of reading and navigation with a minimum of resizing, panning, and scrolling — across a wide range of devices (from desktop computer monitors to mobile phones).
The designer creating a Responsive Design should ensure that the website’s navigation elements, screen-layouts, text, images, audio/video players and other UI elements re-adjust themselves on a variety of devices. Thus, one need not spend extra time and money in creating and maintaining one “mobile-site version” and another “desktop-site version” of her website.
“Responsive Web Design is a collection of techniques that allow a website to flex and adapt to the size of screen it’s being viewed on. Someone opening your site on a small smartphone will be shown the same site as the person opening it on their laptop but the site will have noticed the constraints and automatically reformatted to give the user an experience better suited to their device. No more loading a huge website and having to zoom in and out to find the content you're looking for. Responsive web design takes into account interaction too and makes your site easier to use by acknowledging and integrating things like touch screens to aid navigation.”

Why is Responsive Design important for websites?
As more people are beginning to use mobile devices, like smartphones and tablets, for every task that used to be only capable on desktop, one thing has become clear: mobile is taking over Internet surfing. And, it's not even just surfing. It's everything from browsing social media outlets, checking emails and doing some online shopping.
Because mobile Internet usage is increasing steadily, it's extremely important that your website is mobile friendly. Usually this isn't a major concern. You have a website designed for desktop users and another site specifically developed for mobile users. But, is it possible to have a site that is equally favorable for both desktop and mobile users?
There actually is a design that can handle both types of users. And it's called responsive web design.
Time & Money
The notion that making a responsive website is expensive is just that, a notion. The fact is, while the cost to make a responsive website is somewhat more than making a conventional website, but the expenses to duplicate a website for mobile and other devices gets completely eliminated, as a result – that cuts total development costs, significantly. In addition to that, a responsive design cuts the total ownership cost, by means of taking away the effort to maintain different versions of a website i.e. a “desktop-version”, a “mobile-version”.  Thus, in the long term, investing in responsive website design is the smartest decision.

Pervasion of the Mobile Devices
Internet traffic originating from mobile devices is rising exponentially each day. As more and more people get used to browsing the web through their smartphones and tablets, it is foolhardy for any website publisher to ignore responsive web design. The “One Site Fits All Devices” approach soon will be the norm.
User experience
While, content is king and discoverability of content are foremost success metrics; it is the user experience that enables visitors to consume content on any website through the device of their choice and preference, anytime. Thus, responsive web design is about providing the optimal user experience irrespective of whether they use a desktop computer, a smartphone, a tablet or a smart-TV.

Device Agnostic
Responsive Websites are agnostic to devices and their operating systems. A responsive web design ensures that users get the best and consistent experience of a website on any device of the user’s choice and preference – be that the iPhone, the iPad, the smartphones running the Android OS, or the Windows OS and several others. As a result website owners and content publishers can need not exercise the option to build versions of their website for every popular device platform which they expect their audience might be using.

The way ahead
Thus, rather than compartmentalizing website content into disparate, device-specific experiences, it is smarter to adopt the responsive web design approach. That’s not to say there isn’t a business case for separate sites geared toward specific devices; for example, if the user-goals for your mobile content-offering are limited in scope than its desktop equivalent, then serving different content to each might be the best approach.
But that kind of design-thinking does not have to be our default. Now more than ever, digital content is meant to be viewed on a spectrum of different experiences.

I wonder what sort of device you’re reading this on. A laptop? Maybe you’re at your desk reading the words on a cinema-sized display. Or - perhaps even more likely - you’ve got a smartphone in your hand and you’re thumbing through on your way to another meeting.
When you look around at the incredible plethora of devices on the market these days it’s not surprising that current market research shows mobile internet usage is set to grow massively by 2015 with an increase of 16% from 2010. What is perhaps more interesting is that this means the number of mobile internet users will overtake that of desktop users during this time.
Undoubtedly this change in the way people are consuming online information and viewing websites means we need to be considering how we engage with them to offer the best online experience for our brands and services. So what can we do to make this happen?
When mobile phones started to become internet enabled (remember WAP?) people wanted to access certain information on the move. Checking train times, the weather, or keeping up with the footy scores on a standard desktop site meant difficult navigation, large slow downloads big bills and an all-round poor user experience.
Many businesses realised the need to cater for mobile users, and until fairly recently many opted for a separate “mobile” version of their website. You may have notice a few yourself. You know, the ones starting with “m.somethingorother” instead of “www”.
These mobile sites are completely separate from their larger desktop siblings and have reduced functionality and content. This toned down version means users are only able to load and use the very basic of site content and functionality, and ultimately miss out on the information they may be expecting to find. Separate mobile sites come with logistical problems too - two sites equals two things to design, build and manage. That means ongoing cost and time implications and all for a less enjoyable experience for the user. Not a great solution is it?
Another issue with mobile sites is what mobile device do you build them for? There are too many different screen sizes and differing capabilities to do them all so which do you choose? How many would you need?
Well, with Responsive Web Design the answer is easy.
·         Future proofing. Responsive sites work well across the multitude of existing devices on the market. It’s a safe bet it will for considerable time to come.

·         Better, faster, smarter user experience. Optimizing your site no matter what the user chooses to view it on makes their life easier. Happier customers mean a happier business.

·         Cost Effective. Responsive sites take a little longer to put together, but they survive longer and the unified approach means management, support and upgrades only need be applied to one place. That saves time and money.

·         SEO optimised. Managing SEO for separate mobile and desktop sites is hard and doesn’t produce great results.

·         Improve conversion rates. An optimised and consistent site, no matter what platform it’s viewed on, provides a better experience for the user which is more likely to lead to them engaging with you than going elsewhere.

… And Do You Need One?

1) SOME MORE LIGHTS ON  responsive website?
A responsive website is a site that restructures and reorganizes itself based on the type of device someone is using to view your website (desktop, tablet, mobile phone, etc.).
Quite simply, a responsive site adjusts for different-sized screens.If you have a non-responsive website, your site always appears exactly as it would on a desktop—which is great on a desktop, but not-so-great on a mobile phone.
On a phone’s small screen, your site is teeny tiny, making it very difficult to use. 

View of non-responsive website on a desktop:

View of the same non-responsive website on a mobile phone:

When a site is built responsively, these problems are solved. They display content in user-friendly formats, regardless of the device that’s being used.

But how do I know if our business needs to upgrade to a responsive website?
There are several steps you can take to figure out if responsive design is something you should consider:
1.     Open up your website on a phone. 
Click around to see if you can easily read the text and open links. Is it difficult to use your website? Can you easily see and click on your company’s phone number and have your phone make a call?  Is it easy to fill out and submit any contact forms in the site?

2.   Check out your website analytics. 
Look at the percentage of people who are visiting your website on mobile devices. Is it 10 percent or higher? Are you potentially frustrating (or worse, sending away) 1 out of every 10 of your site’s visitors?

While you’re reviewing your analytics, also make sure to see if this percentage has been trending up over time (six months, one year, and two years). Are you seeing an increasing number of mobile visitors?

3.     Think about your target audience.
Do your target customers actively use mobile devices to look at websites? If your business serves digital moms in their 30s who use their phones to stay connected to the world, that’s the audience that matters—not your company’s employees, not you’re CEO, etc.)
The good news... and the bad news
The good news is your business may not need a responsive website right now—even if you answered “yes” above.
We have clients who we’ve advised to actually stay with their non-responsive websites.
The reasons vary.
Sometimes, their mobile percentages just don’t justify it. Other times, their target customers just don’t use mobile devices to research their types of businesses. And other times, they have a complex website that would exceptionally costly to change.
Bottom line: Upgrading to a responsive website wouldn’t make the best use of these clients’ time and money.
And that leads me to the bad news: Time and money.
There is typically a significant amount of work that goes into upgrading a non-responsive website into a responsive website. It often means reworking the code of your site from scratch—a time-consuming task—and it’s rarely cheap.

Responsive web design offers the way forward…..