Spider Pageview and Pageview Suspects

Last post 10-23-2009, 3:52 PM by Murdock. 11 replies.
Sort Posts: Previous Next
  •  10-20-2009, 8:16 AM 2327

    Spider Pageview and Pageview Suspects

    I am not that familiar with either of these terms and was wondering if I should be concerned if there is a spike in either or both of these.

    Suzie Carter
    www.mountain-cg.com
  •  10-20-2009, 3:25 PM 2329 in reply to 2327

    Re: Spider Pageview and Pageview Suspects

    Oddly enough, I came here looking for almost the same answer.  My Pageview Suspects have spiked out through the roof, but my sales have plummeted in the past 5 days.  October is typically our slowest month, but it has me wondering if there's a correlation somehow.  I don't know what Pageview Suspects are, so I came here to find out.
  •  10-20-2009, 3:38 PM 2330 in reply to 2329

    Re: Spider Pageview and Pageview Suspects

    Spider Pageviews come from identified web crawlers like Googlebot, or ScanAlert if you use that (now known as McAfee HackerSafe).  You can actually add your own identifiers using the Analytics Rules.

    Suspect Pageviews are typically automated scripts being run against your site in an attempt to find a known bug in common web site software.  This could be from a "zombie" PC infected with a virus, or some bored teenager in a far away land.  They can also be unidentified spiders.  Suspect pageviews are identified by a single IP address starting many new sessions very close together.  This type of activity is highly "suspect".

    We seperate these out because if they were included in the normal Pageviews / Session statistics it would make the data meaningless because one Googlebot visit would appear like 100's of unique visitors hitting the site for one page each.

    If you determine that a specific IP address is bombarding your site (look at the Visitor Logs and turn on Show Suspect), you can block it via the IP Firewall.


    Stefan Barlow
    Co-Founder, VP of System Operations
    AmeriCommerce
  •  10-20-2009, 3:51 PM 2331 in reply to 2330

    Re: Spider Pageview and Pageview Suspects

    Thanks Stefan!  Clearly it has nothing at all to do with sales, though it might help me find something eating up bandwidth, correct? 
  •  10-20-2009, 3:58 PM 2332 in reply to 2331

    Re: Spider Pageview and Pageview Suspects

    That is correct. Typically a spider or suspect pageview won't be downloading images but the HTML being sent can add up over time for sure.
    Stefan Barlow
    Co-Founder, VP of System Operations
    AmeriCommerce
  •  10-20-2009, 4:08 PM 2333 in reply to 2332

    Re: Spider Pageview and Pageview Suspects

    Some other facts for the public:
        - Suspect Sessions are is still logged as a full session so it can increase your database size and footprint.  
        - Spider Views are tracked at a less detailed level saving space and database operations.
        - Full Sessions (normal visitors): every page and detail about their visit is tracked for your reports and research.

    AmeriCommerce knows it is a spider by following the Traffic & Analytic rules setup in the admin under global settings.  We give you the ability to override or add to our default rules to classify and more importantly block or redirect traffic that you have researched.   It's a very very powerful module that gives you fine grained control over your visitors.

    Be sure to check it out!  It can reduce your bandwidth and give you great new ways to track where people are coming from such as affiliate sites, emails, branded traffic (people who type your name in directly...that way they are not considered 'organic search' traffic), etc etc.   You can even control where they 'go' too.  There are many 'actions' you can perform on each visit.

    As for bandwidth, I always like to recommend blocking international spiders if you do not sell internationally, this can reduce bandwidth significantly as some international spiders are not very optimized and can consume a lot of bandwidth on your site. 


    Ed Sturrock
    Co-Founder, VP of Product Management
  •  10-20-2009, 6:49 PM 2335 in reply to 2333

    Re: Spider Pageview and Pageview Suspects

    Hi Stefan,

     Thanks for the info.  Is there any instructions or additional information on blocking how to block the international spiders you mention?

  •  10-21-2009, 6:56 AM 2336 in reply to 2335

    Re: Spider Pageview and Pageview Suspects

    Wow, thanks for all the great input. Keep it coming.

    Thanks.


    Suzie Carter
    www.mountain-cg.com
  •  10-21-2009, 1:43 PM 2337 in reply to 2336

    Re: Spider Pageview and Pageview Suspects

    Yes, the advanced analytics rules used to block certian spiuders was a KEY thing for us. By blocking certain russian spiders and other search engines, we were able to cut our bandwidth bill in half ;)

     


    Article on Me & Americommerce
  •  10-21-2009, 1:58 PM 2340 in reply to 2337

    Re: Spider Pageview and Pageview Suspects

    I typically will take a look at a customers site and see what 'unknown spiders' are hitting it.  I also look and group by user agents (thats where the spiders typical information is at).  All of this is done under the Visitor Analysis report which is REALLY powerful.  Just pick user agent to break the visits out by and it will tell you how many hits they are putting on your site.  

    Any known spiders to AmeriCommerce will be in the spiders report so you will be able to see how many times each of them are hitting you.

    That's how you identify the culprits.  Next you will want to block them.  Use the Traffic & Analytics Rules under global settings to do this.  I typically block at least these known international spiders if you don't sell much overseas.

    Baidu - Japan
    Goo - Japan
    Jyxo - Czech
    Sougou - China
    Voila - France
    Yandex - Russia
    Yanga - Russia
    Cuill - US but huge bandwidth hog and not even a live engine

    Those are the really big culprits.

     


    Ed Sturrock
    Co-Founder, VP of Product Management
  •  10-21-2009, 2:00 PM 2341 in reply to 2340

    Re: Spider Pageview and Pageview Suspects

    FYI: The implementation team can hop in and block all of the main international spiders for you for $65...this can save a ton of money over time, in some cases the ROI is < 1 month depending on your current traffic.  Let Manjula Higginbotham know if you are interested or fill out a support ticket and they'll take care of you.  This is the "Block International Spiders" QuickStart.
    Ed Sturrock
    Co-Founder, VP of Product Management
  •  10-23-2009, 3:52 PM 2361 in reply to 2341

    Re: Spider Pageview and Pageview Suspects

    If anyone out there is not sure abou tdoign this, I HIGHLY recommend it, especially for $65.. it will pay for itself very quicly by not wasting bandwith... that is unless you want to expand internationally to the markets which those engines serve
    Article on Me & Americommerce
View as RSS news feed in XML