I have used Linux for quite some time. I just like to have a dev environment that closely resembles my hosting environment.

Regarding Google Analytics, I tend not to use this because for my own uses I can get exactly what I need from the server log files.

Google webmaster tools does have its uses. By using webmaster tools you are simply accessing data that Google already has. With Analytics you are feeding it the data.

Webmaster tools can be great for discovering potential crawl or access problems.

Mack.

I couldn't care less about Google Analytics (seriously, but that's jut me). About tools to manage the sites I've been focusing over the past years on a few things:

• Being able to schedule posts/articles
  
• Optimize seo like uploads having specific names, alt, title, etc
  
• Making backups an easy task
  
• Automated backups
  
• Redundancy, sending backups to diff places
  
• Build site structure to make it easy to recover from disasters
  
• Sites config easy to restore after crash or lost info
  
• Better 404 not found pages with suggestions to related content
  
• Egss on diff baskets
  
• A nice domain manager so I can batch DNS updates
  
• Protect data as much as I can
  

There are tools for that, I ended up writing my own because I like it and I stop depending on other services or fees, but the importance on having a plan is, your traffic depends on it, earnings depends on it and for all of us who focus on having a nice user experience, we don't like users to see broken stuff, it even hurts seo, errors propagate pretty easily. And... because you never know when the problems might come, you might be too busy, traveling, far from a computer, etc. It is important to find good use of cell phones and tablets so you can answer and fix stuff easily too.

As for linux, yes it makes sense to get used to the environment and using standard or popular configs so you can even test locally before going live production.

I know the post is located on Adsense and mine might come as "what?" but as I said, the site being online and healthy means a lot for your earnings. Sometimes I wonder about some discussions because some people don't really have a clue if the site has been online or not, or they are dealing with issues after a crash, it is very important to have a site working consistently before jumping to study adsense stability.

My 2 cents.

AdWords. Seriously. I have always maintained and always will maintain that you cannot possibly get a real grip on how AdSense works unless you understand how the other half (AdWords) works. So if AdSense is worth it, then spend $50 or a hundred bones (or better yet, cop a voucher for free adverting - they're all over the place, but only valid on new accounts) and teach yourself how to run a small campaign in the Display Network.

Or read these:

[support.google.com...]

Or watch these:

[youtube.com...]

But there's really no replacement for running it yourself.

cop a voucher for free adverting - they're all over the place, but only valid on new accounts)

would this be a new Adwords account or would already having a Adsense account be considered unqualified for the vouchers? Does anyone know?

WDR

AdWords and Adsense are considered very different. The voucher for AdWords would be for use with a new AdWords account. It doesn't matter if you currently have an Adsense account.

Mack.

Regarding other analytics tools. One major difference between Google Analytics and many other tools is that you can get IP addresses from a tool like piwik (and many others) but not GA (at least the last time I used it anyway). Being able to associate IP addresses with particular activities can be the difference between screaming at Google in vain and finding the real culprit.

Knowing how to access and use the information in your server log files is also a critical skill to have.

Perhaps this thread should attempt to break down this topic into sub-topics a bit e.g. Traffic analysis tools, security tools (including .htaccess, firewalls, anti-virus, ssl and more), dns tools, throw in methods for blocking nasty bots and, wait, I think we're talking about the whole of webmaster world here.

Take a look at this entire site if you want to learn how to be a webmaster. Think this stuff doesn't apply to you, think again. Paying for a hosting account, auto-installing a copy of Word Press writing some articles and slapping Adsense on your site does not mean you are a "webmaster." It means you know how to push buttons and maybe write a bit.

The reason so many sites are hacked these days is because there are a whole lot of website owners who don't know anything about running a website. What they understand is how to log into their website and post content and place ads. There are many levels of knowledge out there but very few are truly competent in Internet Security these days. Most of us are somewhere between novice and Professional. That means this thread is a worthwhile one if it get's even one person to tighten up their act just a bit. We live in an age where mega hosting companies would like us to believe that anyone can run a website because that boosts sales but I can't tell you how many website owners I've met that don't even understand simple things like the fact that Internet Explorer and Mozilla Firefox are something known as a browser. To them it's just a button on their desktop and if you move it off the desktop, they'll never be able to find the Internet again.

Not pointing at anyone here. Just making the point that we all have something to learn and if you want to be in this business, now would be a good time continue expanding your knowledge because what you knew yesterday may already be obsolete. What you never knew, could likely sink your boat.

I for one couldn't have known for sure where some of my bot problems were coming from were it not for my AdSense-connected GA account. I run Piwik, so I found the IP numbers there, but I couldn't have *confirmed* it, because nothing else will report AdSense that way.

(I know all the arguments about GA, and they're perfectly valid, but not a priority for me)

I for one couldn't have known for sure where some of my bot problems were coming from were it not for my AdSense-connected GA account

I think I've heard you mention before that you don't use the Adsense interface where reporting is concerned netmeg but could be wrong so don't quote me. ;)

Anyway, I'm wondering if using something like piwik in combination with standalone Adsense stats would have provided the same confirmation. I'm sure there are at least some here that don't have Adsense and Analytics linked up and may not even be using GA. Just wondering if GA made it easier or if it was the only way someone could arrive at the conclusions you did.

With Piwik, I have everything I need except the AdSense clicks and earnings.

With GA I have everything I need except for IP numbers.

So for me, both are necessary. I can tell which pages the bots visit from Piwik, and then confirm that those are the pages that are getting clicks (and then losing them) in GA.

As for other tools/skills, publishers should have a grip on. Where blocking is concerned, I think it's important to understand your options and how to go about it. Granted, if you're running a php-based CMS on a shared Linux hosting account, your options are going to be limited. Having said that, here are some options you may have available to you depending on your hosting configuration.

Routers and/or hardware firewalls (if you have access to these, lucky you). These devices can stop traffic before it ever gets to your server (saving bandwidth and server resources, and preventing bots or people from ever getting near your content). This is the expensive route but hey, if you have something you really want to protect and it's making enough money to justify the expense, this is the way to go.

Software firewalls. These generally are going to use server resources but again, can keep unwanted traffic away from your content. You're liable to find this option available by default or as an add-on with dedicated server packages. The trade off here is that the firewall is going to consume server resources so you'll have to think this through before using to block everyone outside the US for example. That can be a lot of rules and processing overhead.

Next, depending on the type of application you're running and what kind of box it's running on, your application can be used to block traffic. On an Apache server, you'll have access to .htaccess and on a Windows server running .NET applications, you're site will have a web.config file. These two file types can be used to control access to you're content as well. Again, with both types, the use of server resources is involved. This is going to be the most common approach available to most websites living on a shared server so it's the approach most widly used.

Of course, there's also the option to write banning procedures directly into your code but this means the unwanted visitor is actually running code on your website even if the result is that they end up on a 403 page so more resource usage and you need to know something about programming to get it done or use some third party script and hope it does what it's supposed to.

Regardless of your approach, there are some fundamental things you need to be able work with if you want to block by IP address or IP Range and this is where my recommendation begins.

Whether you're creating a rule in .htaccess or on a hardware firewall, you need to know how to specify IP range more often than not. A critical skill in this task is knowing how to specify an IP range using an IP address and a subnet mask or using CIDR notation.

Having said that, this is a fairly complex subject that at first blush looks to apply primarily to network administrators and, yes, it was probably a network admin who set up the subnet you're trying to block. What you need to know is how to use subnet notation (or CIDR notation) so you can block that subnet. So, I'd recommend becoming good friends with a subnet calculator (just found a good one searching on DuckDuckGo so you can too) and doing a little reading on how subnet masking works. This falls under the category of a fundamental skill for webmasters IMHO and opens up the opportunity to put that skill to use in defense of your Internet property.

With Piwik, I have everything I need except the AdSense clicks and earnings.

With GA I have everything I need except for IP numbers.

So for me, both are necessary. I can tell which pages the bots visit from Piwik, and then confirm that those are the pages that are getting clicks (and then losing them) in GA.

I have to agree with this. GA and Piwik work well together. I have customized my GA "dashboard" to reflect domain names and associated clicks. And I use Piwik to determine IP addresses and the patterns of reading. Piwik is a remarkably useful tool that gets better with every iteration.

And I am investigating Nagios, a commonly used network management tool, to see if it will improve my ability to spot which visitors are doing click-fraud.

On edit: I should say here that Nagios is not for the faint-of-heart. It's major claim to fame is that you can run it remotely (from your home or office with a static IP) and just run a client on the web server. This reduces server overhead.

WDR

I would like to try Piwik but their demo does not show IP addresses. Just a 2 question for the people which use Piwik

a)
Is possible by Exit pages see IP addresses associated with pages? Something like statscounter show up?

b)
I read "if your websites have more than 10,000 pages per day, you may need virtual server or dedicated hosting"

So what's is your experience here with effect of Piwik on the server?

My website hosted on semi-dedicated server have between 30.000 - 40.0000 pageviews per day with 10.000 - 13.000 visits using server with 8 GB memory, 8 core CPU (per core 2.27GHz) and MySQL 5.5.36-cll.

I think.... If i should have with Piwik anything that i need comparing with Statcounter then why to pay Statcounter if i can have with Piwik everything for free.

The Piwik demo does not have IP addresses to protect the people who visit and check it out. But the real Piwik does show IP addresses; and will track a visitor across 100 visits. Piwik shows you every page a visitor looks at as a link so you can click on it and see which page it is (handy for a forum).

It shows exit pages but I don't see IPs for exit pages. You could glean that from the last page read for a visit, though.

At my peak in the summer I run about 30,000 page views a day and Piwik doesn't seem to have a problem. However, if I want to look back deep into the database it gets pretty pokey.

My hosting provider moved me to a better machine as my traffic started to pick up. I don't know if I'm "semi-dedicated" or not. We have fiber to my office (although I'm using DSL now) and if I need to I can set up a dedicated server right here.

I don't use statcounter but between GA and Piwik I think I have pretty much everything I need. All I would want beyond this would be profitable enough to host everything right here with two routes out.

WDR

My systems are dependent on Adsense income... So I am sure its best to use Google Analytics, Webmaster Tools, Adsense.
It really gives you a overall picture of your site.

@Howzitza -- Where do you get IP information from? How do you identify a troublesome visitor/bot?

Very intererstin Question.
Its a combination of using what server you are on, your dedication and Analytics Real-Time system.
It also depends what the troublesome visitor are doing.
If its argument sake a spammer trying to flood your "Contact Us" form or comment system, you can:
1. Add a column to MySql table to capture the actual ip address of poster
2. Follow the user on Google Analytics Real-Time system and get the ip from your "Latest" visitor stats in your cpanel,

No need to buy expensive systems to identify a troublesome visitor. But all said... It depends on your individual system setup and the exact problem at hand.
It also depends on how smash pumpkin nerd you are...
(I am one of them... Laptop stuck to my face, living with my system...)

Thanks for response wa desert rat.

I will probably a wait a bit before taking decision to install Piwik.

If there by exit pages are IP addresses visible then i would probably try to see how it perform with my server but right now i will wait a bit. Maybe they are gonna to release it next version of Piwik.

I use very often Apache logs in combination with program weblog storming. It is good program for analyzing raw apache logs. Then GA and recently started with Statcounter.

What i like by statcounter are exit links with format of Adsense, clicks and IP addresses.

Actually not any other program show that in one place. Statcounter cost money but unless i can find something else i will coontinue to use it.

And probably i try to find out of there is some coder who can fix Asrep analyzing program. It was great program just to show you what happened with Adsense including clicks, IP addresses, countries and anything else.

Statcounter... Cool program to use. Or it was for me about 10 years ago. I stopped using it when I found that some Anti-Virus and programs like Spybot S&D marked it as Malware infection cookies... Not sure if its been cleaned?

Statcounter... Cool program to use. Or it was for me about 10 years ago. I stopped using it when I found that some Anti-Virus and programs like Spybot S&D marked it as Malware infection cookies... Not sure if its been cleaned?

I have not had any luck independently sniffing clicks. I have not been able to make Piwik do it (exits are shown but not correlated to ad clicks).

I'm thinking that Nagios might be better able to see click traffic to Adsense but I'm not terribly optimistic. I haven't tried statcounter. Being able to see what domains executed clicks via GA is probably good enough anyway but all those invalid clicks made me much more interested in more granular analysis.

The fact that Piwik was free and, at the time, I was trying to identify some mystery traffic on my main site were the factors that moved me towards that utility. I have to say that updates - which always scare me - have been automatic and painless. But it does put a strain on the hardware.

Although I can ssh into my server, not having control over traffic flow and hardware is a significant handicap.

WDR

When the click-swarms were going fast and furious I started logging my Performance Reports in a handy (and free!) "note" application called "Cherry Tree". This app is available for Windows, Mac and Linux and what I like is that I can just copy the data and then paste it right into CherryTree. The columns don't always line up right but it's easy to edit it for more descriptive notations. Seems quick and gives me the ability to more closely monitor the performance history of my ads. Or non-performance, as the case may be.

A review: [makeuseof.com...]

The site: [giuspen.com...]

WDR

StatCounter exit clicks are NOT reliable as regards AdSense. Do not count on them. StatCounter says so and Google says so, and my own anecdotal evidence over several years and many sites says so.

Netmeg - where Stascounter says that exit links are not reliable?Can you please give to us more informatio about it?From my point of view Statcounter works very wekl and show much more clicks than Adsense.It is not strange because Adsense show onky valid clicks.

They said it when they announced the Exit Link option, some years ago. And no, I'm not going to go research it for you. I know from my own sites that it is NOT reliable. At one point it showed exit AdSense clicks on a site for my own IP when the site was not even live to the world yet; I know for a fact that I was not clicking on ads, nor was my developer when it showed them for his IP. And frequently I get clicks in my AdSense account that never show up in StatCounter exit links. Occasionally it gets them right, but I certainly wouldn't rely on it in the least.

Thanks for explanation but if they did make Statcounter as webtracker which analyze visits then i think that everyone should go away if they know that creator of Statcounter said 'we did it make it but it is not reliable'. What's point of having some option if it show fake result

But anyway thanks for sharing your experience based on this tracker.

btw

I did quick research but could not find any official statement of Statcounter regarding this matter.