We've heard a comment from a number of "A-Listers" that all of your social media activity should be "organic." That you shouldn't use automation or do anything that isn't natural with sites like Digg, because somehow that's cheating.

Our question back is "What's so noble about carpal tunnel syndrome?" If you're using automation to do the same things you would do if you sat at the computer and clicked your mouse in a mind numbing session, what's the big deal? Is it the user's fault that most of these social networks have built such a brain dead and poor user interface that accomplishing even simple tasks requires minutes of clicking? Is it the user's fault that these sites take forever to load a page? Is it the user's fault that most of these sites have limited their APIs to the point of near uselessness?

What's the difference between using a script to accomplish simple tasks and hiring an intern or someone from India to do the same thing? Is an agency that pays a staff of people to mindlessly click on social sites all day somehow more magnanimous than the sole proprietor that uses a macro to accomplish the same thing? Do you really think that these big name consultants that charge several hundred dollars for a consultation are spending hours a day of their own time clicking their mice on social networks?

How about the use of a programmable keyboard or mouse? What's the difference between using a FireFox plugin like Greasemonkey that will click a few links for you and open a window and using a programmable keyboard that will do the same thing when you hit a function key?

How about using a browser plugin that automatically opens all links on a page in a new tab so that you can easily read the stories? That's a script, isn't it?

How about using a super high-speed internet connection? Doesn't that give you an unfair advantage over someone using dial up? Doesn't that put more of a load on the website? To be fair, shouldn't you be using a 56K dial up line to do your social networking?

If social networks would make simple tasks like unfriending 100 friends that are completely unproductive a one step operation, there wouldn't be a need to use scripts. Instead, they're trying to maximize page views. So frankly, using scripts helps them with that goal. If it weren't for automated scripts, a lot of their traffic and revenue would go away.

Do as They Say, Not As They Do

Don't think for a minute that the majority of the people pontificating about "organic" use of these sites aren't using automation themselves. We're not going to call anyone out by name, but by using our tool to analyze the friendship networks and voting habits of Top Diggers, it's quite clear that they're very automated and voting in blocks like crazy. Search your feelings, you know it to be true.

If you buy into the party line that only "organic" use of these sites is acceptable, you're just guaranteeing that you won't be able to compete with the "A-Listers" -- which, frankly, is exactly they way they'd like it. They're not in business to create competition. So if you'd always like to be in second place, listen to their advice.

I'll also point out that the same people that rail against using scripts to make social networking more efficient usually have installed ad blocker software. So who is really stealing bandwidth?

Won't I Get Banned?

True, if you do something crazy and Digg a few thousand stories in a few hours you're going to get caught. Especially if you're testing a script in development and it gets away from you! Not that I know anyone that has happened to or anything. But if your script has the appropriate pauses, you only vote a percentage of stories, and you do the same things a human would do then there's a very low probability of getting caught.

And frankly, you shouldn't be putting all your eggs into a single basket. Keep an offline record of your relationships. If you get banned, it's just a matter of a new IP address and adding back the relationships you've already built. Your tools can do that for you, can't they?

Keep in mind that the top users get forgiven when they get caught using a script. I've heard several instances of top diggers that got banned and reinstated the next day after promising to behave. They're still misbehaving; they're just doing a better job of not getting caught. But if you're not one of the top people, good luck even getting your emails answered.

It's How You Use Your Tools That Count

The same car that can be used by a drunk driver at 2:00AM can also be used to drive the kids to school at 8:00AM. The car isn't good or evil -- it's how it's used that matters. If you're using tools to spread spam the community is going to punish you anyway, so it's not going to work out. But if you use a tool to be more efficient and do the same things that you'd do anyway if you were willing to risk carpal tunnel syndrome and spend 10 hours a day clicking then there shouldn't be any problem.

The real question is "How much is your time worth?" If it's not worth $19.95/month to save a few hours a day of your time, then keep clicking. Otherwise, you really ought to look into automating some of your drudgery.

We did the Imacro Script to Add the Problogger Digg Friends the Easy Way earlier this week.

Now we've done the same thing for the giant 538 Twitter Users That Blog list that he put together.

You still need to install Imacro, but this time to make it easier you only need the FireFox version. You can click on the Imacro icon in our sidebar. Then just run this script. Be sure that you're already logged into Twitter before starting the script.

Thanks to Darren for putting this list together. This script will save you a lot of mouse clicks.

Darren Rowse at Pro-Blogger has collected Hundreds of Bloggers to Interact with on Digg and StumbleUpon. It's in the form of a bunch of links that you can visit and hit "Add Friend". That's a great list because these are all people who will most likely be interested in your message.

I suppose you could spend several hours adding all those friends and develop a good case of carpal tunnel, or you could just use an Imacro Script to Add Those Digg Friends. It will run through that entire list and click the "Add Friend" button for you.

If you don't have Imacro installed, go do that right away. You can click on the Imacro icon in our sidebar. Then just click the link to the script or download it using right click and run it in Imacro. If you don't know about Imacro, you're about to enter into a wider world.

Some people have expressed reservations in the comments over at Problogger that adding a lot of friends at once in Digg or StumbleUpon could get you banned. I'm not sure what they're basing that on, but our experience from dozens of people over several months is that Digg doesn't mind adding lots of friends. We're the ones who invented the Digg Friend Finder and now the PMS Social Suite, so if banning was going to occur we'd have seen it. YMMV.

Participating in voting blocks and gaming the system will get you banned, that's been documented. Adding lots of friends, on the other hand, is what those sites are designed to support.

I'd like to thank everyone for their testing on the Social Suite with Digg Analytics and Automation. It was, well, interesting. I've done a lot of beta testing inside client sites but haven't really released a program into the wild since before the internet. (Remember FidoNet and shareware distribution? Yikes!)

At the end of the day we decided that it was usable enough to replace the old Digg Friend Finder. Which, given the number of daily users, was a pretty uncomfortable and tricky decision. However, the actual traffic on the free version of the Social Suite has gone up from the old Digg Friend Finder, so mission accomplished there.

Mainly, however, it was quite a learning experience for us in how people use automation software when it's not part of a larger corporate sponsored project.   We clearly recognized that individual or small company buyers had different price and function points, but the variable cost of time and overhead is so much less accounted for in smaller firms that a lot of our positioning was probably not necessary.  Fascinating. 

Social Suite Beta Test Pre-Natal Expectations

I was expecting a LOT of criticism for the UI. I quite like it and it has a lot of technical advantages from our standpoint, but it is not the typical UI.

Not one word.

I was also expecting people to balk at installing iMacros (especially since you have to install the previous version because of bugs in the new release) and running the Suite in its own Firefox window.

Not one word.

I thought there were too many columns of numbers in the Find Friends panel for people to really wade though them. It turns out people ignore the numbers they don't understand or think are unimportant. Fascinating.

We got a lot of good feedback about our documentation and how to help reduce the complexity of what the suite can do. Over the next week or so I'll be publishing some articles to help people use both the free and premium versions of the software.

Beta Test Post-Mortem

This is going to sound strange, but our take homes were:

1. Our target market uses paypal rather than Amex. We were startled.
2. People want videos rather than user manuals. I guess it's the YouTube phenom coming home to roost. I actually find it easier to write a manual. (Yes, I own a typewriter, why do you ask?)
3. We were right to go with value based pricing and to aim for "professional diggers."

Value Based Pricing

There are (broadly) two ways to price anything: cost or value. Walmart prices own-brand cornflakes a price+markup. Apple prices everything at value. The difference differentiates your market.

So when we decided to price the first version of the social suite we tried to balance off users, user time/value, revenue, support costs/expectations, server load, investment timeline, etc. We put a stopwatch on a lot of in-house testing, spoke to the alpha users extensively about the value proposition, and did some magnificent fiddling on a whiteboard.

And came up with a buck an hour.

If you value your time at more than a buck and hour and use digg to drive revenue, then you should be paying us to use the social suite. And our beta testers, as they converted to paying customers, confirmed this observation.

Why Not Charge More?

If you look around, there aren't that many SEO tools that successfully charge an admissions fee. So our goal was to establish a precedent and, as we add value and reduce time/cost we will raise our prices.

If You Missed the Beta

Look, if you were taking a nap under a lilly pad or something, just go to our contact us page and drop us a line and we'll help you out.

Wow, that may be the most specific post title I have ever written.  But I want to capitalize on the extreme amount of google love we're getting lately to save someone else a LOT of time.

We're doing a beta of our Social Suite and it depends on iOpus's most excellent iMacro for FireFox.  Except that they released a new version (6.0.4.1) which killed our software.  Thanks guys.

So, how simple could this be: we just needed the previous version (6.0.3.9). 

Which is NOT on their website.  Nor is it mentioned in their otherwise comprehensive wiki. And when you try to search google for it, well, good luck with that. 

So, it turns out that previous versions of iMacro for FireFox are kept on the FireFox/Mozilla website.  Go here to get version 6.0.3.9 and earlier versions of iMacro for FireFox.

You know what else I know now?  There is (often) a "see all versions" button at the bottom of the addon for FireFox that gets you to the same place. 

If you would be interested in beta testing our new social media automation product before it is released, please drop me an email.

You should be

• An experienced Digg user
• Wiling to install iMacro for Firefox
  
• To spend at least a few hours using the product (My guess is that you will, like I did, fall in love with this product, so it shouldn't be too painful.)

We're very excited about how this will accelerate your ability to be successful on digg.  And if you're of an analytical bent, you will love what you can find out about your (and others) friend networks.

I'm not talking about evil, low-life scraping that involves stealing content.

Data Wants To Be Free

In this case we're looking at how to get data out of an existing system that has nothing more than a web interface. Perhaps for political reasons your IT department won't give you SQL access to the database, but you really need to pull out a copy of the product list and get it into a local MySQL database. Or perhaps you're just trying to keep track of your competition's pricing. Maybe you want to prepare a demo for a client before they love you and give you their product list.

There's absolutely no reason that data has to be locked away and keeping you from getting your job done.

ETL Can Do The Task

Or maybe not.  There are a ton of commercial and free Extract, Transformation, and Load (ETL) tools out there. But they all have one thing in common: they were built for handling large movements of data between a myriad of data sources. But very few of them were built with web scraping in mind.

iMacro Will Save You

Most people with an understanding of Imacro realize that it does a very good job of the Extract and Load processes. But out of the box it's not really very good at Transformation. Imacro is a declarative language, much like HTML. There are no constructs for logic processing such as if-then-else or applying functions to data. In fact, if you want to do anything other than some pretty generic loop processing you usually have to resort to the use of a scripting language.

There's a big problem with relying upon client-side scripting to handle your processing -- sometimes you don't have access to the client. Take our Backlink Pinger, for example. This is an application that requires no other software than the Imacro runtime to be installed on the desktop. The server generates an Imacro script for the client to run, and the client browser detects that it's a script because Content-type is set to application/octet-stream and the extension is .iim. So the user presses a button on a web page and presto they're running an Imacro script. There are lots of situations where you couldn't just install a client side application on the desktops that would need to run it. Imagine the hassle of maintaining a 1,000 desktop installation of a little Visual Basic application that pulls data from the corporate mainframe into a spreadsheet.

iMacro Transformations

So how can you perform transformations in Imacro without using the scripting host? The trick is to use a web page that you control to perform your transformations. You can extract data from one web site using Imacro, submit it to your own site, perform the transformation, then return the data in an easy to digest format for your Imacro.

Let's take a straighforward example. Our task is to write a script that can check Technorati for our current authority ranking and insert only the number into an application that tracks our authority. First, the Imacro to grab the authority:

VERSION BUILD=6120228

TAB T=1

TAB CLOSEALLOTHERS

URL GOTO=http://technorati.com/blogs/promote-my-site.comblogreactions&&HREF:http://technorati.com/search/promote-my-site.com EXTRACT=TXT
SET !VAR1 {{!EXTRACT}}

iMacro and PHP

Running that script places "Authority: NNN" in the !VAR1 variable. But our application needs to strip out the "Authority:" from the string. If you've got access to a php server, then you could write something such as:

echo "<data>". str_replace("Authority: ","", $_GET['authority_string']) . "</data>";

Assuming we save that snippet in a file named "authtrans.php" on our server, we can add the following to our Imacro script:

URL GOTO=http://promote-my-site.com/authtrans.php?authority_string={{!VAR1}}

TAG POS=1 TYPE=DATA EXTRACT=TXT

SET !VAR2 {{!EXTRACT}}

The value of !VAR2 will now be equal to the authority number, without the troublesome prepended string.

iMacro and Javascript

That was a simple example, but the technique can be used to perform wonders. There's really no limit to what you can do inside the tranformation step. Don't have access to a PHPserver? Then just create an HTML page and use javascript to perform the transform. For example:

<input id="aus" name="authority_string" value=""
       onchange="document.getElementById('aus').value = document.getElementById('aus').value.replace(/Authority:/,'');">

iMacro and Javascript - A Test

Fill in this field with "Authority: 123", tab out, and see what happens:

Just use Imacro to fill in that field with what you want transformed, then extract the contents of the field. The javascript to perform the transform will fire and you'll get the modified version back into your script

Imacro is incredibly powerful. If you're not using it, you're wasting your time.

We all know that some backlinks never make it to technorati and therefore are less likely to be "discovered" by google.  And all your backlinks are important for SERPS, high quality technorati traffic, etc, etc.  So we built a tool that will automatically generate pings to technorati and pingomatic for all your backlinks.  And it is free.  (Well, it's ad-supported, but that is close enough to the same thing.)

How and Why

If you'd like an architectural explanation of how it work, you can read All Your Links are Pingworthy.  Here is how to download and install iMacro and why we chose iMacro as a SEO automation framework.

Operations - Getting There

It's quite simple, really.  You can go to our website, Promote-My-Site and click on the left hand side toolbar link for Promote-My-Site Ping.

Note: The really cool screen grab above is from FireShot, which saves me five minutes a day, easy.  Get it if you do a lot of screen shots!

 

Using the Promote-My-Site Pinger

Once you get to the right page you'll see some explanatory text and:

Which is not so useful, so put in your URL and click PingBacklinks.  Go ahead and click through the dialog box telling you that this might take a while - it's actually very very fast but testing feedback said that we needed something like that.  You'll see:

This means that Yahoo had 5,087 backlinks and we are 50% of the way turning those Yahoo API'd backlink records into technorati pings for an iMacro script.  (Now is a good time to download iMacro if you haven't already done it!)

When the process is complete you will see this button:

One of the tricky things here is that Yahoo's API will only let us get 1,000 records, so you may want to split up your website into a set of strong pages, or take each RSS feed and run it through.

Trouble Shooting

Honestly, we have had a great limited beta (aka QA sucker) round and haven't found any problems we can't fix.  If you start to get 0 results for things that you know have backlinks you may have hit Yahoo's query limit for the day.  But the cool thing is that, because this runs out of an Ajax app on your browser, you can just grab a new IP and you're off.

Drop us an email (help@promote-my-site.com) if you have any issues and we'll help.  You can also find us on Sphinn and SEOMoz, so if PM works better for you then you can reach us there pretty easily.  We'd also be delighted to hear any suggestions for improvements and additions.

In order to build an efficient SEO automation framework you must carefully seperate value added manual tasks from reptitious automatable steps. For example, writing content on Social Networking is value added, but posting it to 23 different pligg sites and bookmarking the article at 50+ sites is not.

Manual or Automation

But determining which categories it goes into on the 23 pligg sites is an appropriate manual task. So it's not like you are just going to offshore someone to pickup the phone and say: "Dr. Newharts office, hold please."

Let's posit, for the sake of argument, that you've built a framework of CMS, user administration, content creation, database driven workflow, and iMacro automation of specific tasks. Not a small task, but you can get there stepwise, so bear with me.

What we do here is to flowchart (remember that?) our process and mark boxes as M/A - where:

• M = Manual
• A = Automatic

The trick is to NOT mark something as "M" just because you don't know how to automate it. If you can concieve of something as non-manual, odds are that a bit of sweat equity will allow you to build a tool to automate the process.

Real World Automation Example

We recently had to create a BUNCH of customer records for a system migration, fix address fields, verify zip codes, etc, etc. And by BUNCH I mean it was 200K+ items. We had done this type of work before using Mark V1.0 Humans so we had an idea of times and costs. This time we did a bit of flowcharting and what-if'ing and decided that rather than spend 3 months doing the needful with copy/paste/excel and 40 people we would build an iMacro based toolset.

The tool creation took two weeks, we temp'd a dozen folks to manage exceptions, and were done in 2 months. For half the cost and at a measured 95% accuracy rate.

That is SO NOT SEO

Well, yeah, but did you think we'd tell you our internal SEO tricks? Uh, no.

But use your imagination. Say you have 50 keywords/phrases you buy on Google. Who else buys them? How do they rank in the SERPS? How are their websites organized? What else do they own?

You can certainly figure out all that stuff manually. Or you could write an iMacro to query google, capture the ads, present them to people to decode, send iMacro to run reports on their sites, look up their IP's, etc, etc.

If you Do It More Than Once

You might consider automating. Me, personally, it's only worth getting the geeks involved if I have to do it more than twice/week. Otherwise, in my experience, it may take me more time to specify than to do.

Summary: Automate

I think if I were smarter I'd move my 'automate' bar down to the things I do once/week for more than six months. But, then, I have a fair bit of technical support. If you are a roll-your-own person, your barrier to entry might be even lower than mine.

When in doubt, automate with iMacro and capture your data in a relational database.

The bad news is that if you choose the wrong tool you'll have a heck of a time unwinding the mistake. The good news is that these products are different enough and the choice is pretty clear.

We said earlier that a web application manipulation tool is one that drives a web site based on a link back to your backend application containing workflow and data driven information. In a real world setting this tool must have the following characteristics:

• Supported application - from a commercially viable vendor or an active OSF-type community
• Mature product - must have developers documentation and have deployment successes
• Flexible - Must handle a wide variety of web-based applications

There were only four real world candidates that were close enough to analyze:

• Macro Express
• iMacro
• CoScriper
• AutoIt

I was going to lump Chickenfoot in with CoScripter but I will break it out as it has some particularly interesting academic shortcomings.

I would have dinged both of these tools as not being mature - CoScripter is less than a year old and ChickenFoot is barely 6 months.  I really don't care how smart the guys at IBM or MIT are - that's not a mature product.

But there is another problem with CoScripter, from IBM.  And the problem is IBM.  Normally (unless you remember OS/2!) it is a good thing to buy software from IBM, but it's not exactly in their software strike zone, is it?  Oh,  well, yeah, it's free and everything,  but how does it fit in with their Linux strategy?

It doesn't.  So CoScripter is only as alive as the interest of the researchers working (part time) on it.

ChickenFoot is even worse: senior project at MIT.  Next year, aside from NOT getting the girls, these guys  will be doing what, exactly?  Again, open source, but is that your business?

Here is where the wheels really come off ChickenFoot.  It uses a pattern matching engine to figure out what it wants to click when you say click(“Submit”).  If there are, say, five submit buttons then you have to write a buncha javascript.  Uh, dude, how fragile is that?

AutoIt is in the third incarnation, has an incredibly active community, and receives regular updates.  Best of all, it’s free, small, and looks a lot like visual basic.  And you can call Windows system level ‘stuff’ as well as COM, DOM, and all those other overloaded Microsoft Acronyms.

Which is the real problem:

The web side is pretty much, well, krep.  You can smack mouse click into exact locations in a programmatic window that you overlay on an IE region.  And if that sounds like using a laser cannon to heat your Beenie Weenies, well, it is.

MacroExpress has many of the same powerful windows features of AutoIt but with numerous web features built in.  It is a well supported VB runtime like product, with a relatively active user group and lots of examples. 

It does not handle Java U/I issues, Ajax, Flash, etc.  I'd say that for plain vanilla HTML apps their web automation would work pretty well.  And, yes, I am aware that this is a diminishing crowd.

It costs under $40 and you get about that much support.  The user group/forum seems pretty effective, but there are persistent bug complaints that seem to go unresolved.

This product is several years old, is installed in a host of major corporations and startups.   Of all the products, this is most like tools from 'the old days.'   I was reminded more of MultEdit or WinZip or some other product with a cadre of developers and a wide installed base.

When you buy iMacro (and the developer license starts at $500 and goes up pretty quickly) you get support.  Just like a real product.

iMacro can handle Java, Direct Screen, Ajax, etc, etc.  It can even do fuzzy image recognition of bitmapped objects on screen.  Frankly we've been unable to find a situation where we couldn't  bang on an application using iMacro.

This is probably pretty obvious: iOpus iMacro.  For your amusement, I've placed the candidates on our SEO capability matrix, but I think I can summarize why this really works best: it is the simplest solution.  It has a lot of sophistication under the covers, but a simple glass bottle full of red wine can have a lot of complexity, and history, and artistry too.  So don't be fooled - the buys at iOpus have crafted a specialized tool that eschews the useless and focuses on completing a job just exactly right.

We'll start giving some concrete SEO examples using iMacro and some of the architectural framework we've discussed in earlier posts.