Ernst-Jan Pfauth, at The Next Web, does some back-of-the-envelope calculation and figures out that Facebook may be making $15M/year in selling "virtual gifts." And she smartly urges other social networking websites to jump on that bandwagon.

Anyone remember a slightly less scary looking Whoopi Goldberg hawking Flooze in the last bubble?

How long until our customers want to pay us in virtual credits we can use in Second Life to buy private education for our virtual children?

We put up a post yesterday with a cool table of information about social websites and, because I hate having to scrape stuff off the web, we used google docs to store the information. But that is all we'll use google doc's for - because it has kreppy features and google has, or so it seems, reserved the right to use your "private" documents for whatever they please.

Horrible Lack of Functionality

First the functionality. Let's just say if my options were between Google's spreadsheet and a copy of Lotus-1-2-3 running on an Adds Viewpoint under CPM, I'd probably take Lotus. Yes, their stuff has that little functionality.

For example, some maroon at Google (who I am sure is worth 250x what I am, but never mind that) has breathlessly announced that you can build a web form that saves data in a spreadsheet. No, really. Big News at Google.

Leaving aside that people have been saving web form information to disk for, oh, 10 years ... Excel has been able to create forms since version 0. In 1990.

And don't even get me started on formatting, calculation features.

And a TOS to Bind Them All

But, most importantly, scope this quote from Google Doc's TOS :

You retain copyright and any other rights you already hold in Content which you submit, post or display on or through, the Service. By submitting, posting or displaying the Content you give Google a worldwide, royalty-free, and non-exclusive license to reproduce, adapt, modify, translate, publish, publicly perform, publicly display and distribute any Content which you submit, post or display on or through the Service for the sole purpose of enabling Google to provide you with the Service in accordance with its Privacy Policy.

I can't tell you what our legal consultant said about that because they might think I was giving legal advice and, frankly, I hate it when they come in my office and slap me around. And then send me a bill.

But I will say that we immediately pulled our important documents off google and started up a wiki behind the firewall. Because while I am sure that some lawyer made them put that in there, it sure reads like they can do what they darn well please ... but not with my document full of account numbers and passwords!

I thought it might be useful to compare the sites that the minor productivity enhancers we've discussed and reviewed actually claim to support.

Social Site Overlap

I thought there was more overlap - very interesting.

Your Own Copy Of This List

And because I HATE it when people put tables up on the web and don't provide an easy to get the farking data, here it is on Google Documents.

Reviews Of These Programs

I recently did five in-depth reviews of what I felt were the top lightweight social network posting automation solutions:

• Post Toaster
• Social Poster
• Social Marker
• Only Wire
• EkStreme Socializer

Conclusion

Well, really the same as in the reviews - there is a lot of potential power in there, if only someone would build a scalable, supported system with plenty of workflow and reporting.

Tonight, because of the writers strike, we bring you awards from the world of XML-RPC Pinging:

• The winner of the award for highest ranked blog without posts is ... Ping-O-Matic. (Last post: 30-Apr-2006!)
• And the least coherent description of why data on millions of blog post was lost goes to Technorati which was broken for weeks.  'Kinda.  For a few (cough) million people and a few (cough cough) tens of millions of posts.  Let us know if it hit you.'  (I paraphrase.)

Your Content, Your Rankings, Your Responsibility

I will tell you something you might not have thought about but is important: You are in charge of making sure that Technorati (et. al) know about all the links back to your content.  I won't go into why you care if Technorati (and the google-bot that crawls it all day long) has your backlinks - that is pretty  obvious.  I'm just going to describe the "normal" process and how it might break.

I'll tell you something else you might ought to know: you have to be in charge of your own pinging.

Utterly Normal Post and Ping

In the normal course of events this is what happens:



You post something to blogger (or WP or Joomla) and it causes an XML-RPC ping to go to (at least) Technorati.  Which indexes your page and everywhere you link and gives you love and all your outbound links love.  And then google crawls Technorati and gives mucho love-o.

(Note that outbound links part - it's key later on.)

For Technorati you can substitute Ping-O-Matic and you get 21 sites that accept the XML-RPC ping on a post:



Note that Ping-O-Matic pings Technorati and you may ask why everyone doesn't use just Ping-o-matic?.  One difference is that Ping-O-Matic only accepts domains and Technorati will accept a URL. Pinging the URL can be important because sometimes the spiders may miss your backlink if the depth is too great. Both services will allow you to ping more than once a day. I'm sure there is a limit on their tolerance, but we've done a lot of testing and haven't gotten any nastygrams from either source.

So Simple - What Could Possibly Go Wrong?



Some things we've found that can go wrong:

• Technorati loses the ping (cough)
• Packets are lost in the internet (yes, Virginia, it does happen)
• Your hosting platform says it is pinging but isn't
• You installed Oracle's PHP and the ping simply doesn't go out
• Your content platform simply doesn't support pinging

And I'm sure the list is much much longer.

Real World Example

We had a blog with low posting volume that was getting plenty of inbound links but wasn't exactly burning up the SERPS, but one day, around 200 posts in, we noticed that there was, essentially, zero Technorati traffic.  Which we thought was strange because it should have been geting some traction.  So we checked and it simply wasn't listed as having anything at posted or inbound.

We generated a quick XML-RPC pinger and smacked technorati with all our back posts.  Within 24 hours we were getting traffic.  Cool.

What Else Was Lost?

So we went to Yahoo, got all the backlinks to this blog, and pinged Technorati with them too.  Voila, even more traffic.  Plus our google search traffic went up because of better SERP positioning. Within a few days we had a respectable Technorati Authority score (at least compared to the zero it had been).

Not a Magic Bullet! 

Look, this is blocking and tackling.  And if it is something you can just schedule to do once a month or so then you can make it into a building block. Don was comparing this to making sure that you buy, on average, the least expensive office supplies that can do the job.  But I think that, since we're talking readers or sales, that this is part of what could give you an edge.  and every marginal sale you make helps you become stronger while your competition becomes weaker.

We've put ensuring proper pinging and registration on our checklist.  You should too - the time requirement relative to creating content makes it a no-brainer.

The bad news is that if you choose the wrong tool you'll have a heck of a time unwinding the mistake. The good news is that these products are different enough and the choice is pretty clear.

Real World Web Application Manipulation Tool

We said earlier that a web application manipulation tool is one that drives a web site based on a link back to your backend application containing workflow and data driven information. In a real world setting this tool must have the following characteristics:

• Supported application - from a commercially viable vendor or an active OSF-type community
• Mature product - must have developers documentation and have deployment successes
• Flexible - Must handle a wide variety of web-based applications

There were only four real world candidates that were close enough to analyze:

• Macro Express
• iMacro
• CoScriper
• AutoIt

I was going to lump Chickenfoot in with CoScripter but I will break it out as it has some particularly interesting academic shortcomings.

ChickenFoot / CoScripter

Maturity Test: Failed

I would have dinged both of these tools as not being mature - CoScripter is less than a year old and ChickenFoot is barely 6 months.  I really don't care how smart the guys at IBM or MIT are - that's not a mature product.

Supported Application: Failed

But there is another problem with CoScripter, from IBM.  And the problem is IBM.  Normally (unless you remember OS/2!) it is a good thing to buy software from IBM, but it's not exactly in their software strike zone, is it?  Oh,  well, yeah, it's free and everything,  but how does it fit in with their Linux strategy?

It doesn't.  So CoScripter is only as alive as the interest of the researchers working (part time) on it.

ChickenFoot is even worse: senior project at MIT.  Next year, aside from NOT getting the girls, these guys  will be doing what, exactly?  Again, open source, but is that your business?

Flexible: Too Much So

Here is where the wheels really come off ChickenFoot.  It uses a pattern matching engine to figure out what it wants to click when you say click(“Submit”).  If there are, say, five submit buttons then you have to write a buncha javascript.  Uh, dude, how fragile is that?

CoScripter and ChickenFoot Final Grade: D

AutoIt Maturity And Support– Yes!

AutoIt is in the third incarnation, has an incredibly active community, and receives regular updates.  Best of all, it’s free, small, and looks a lot like visual basic.  And you can call Windows system level ‘stuff’ as well as COM, DOM, and all those other overloaded Microsoft Acronyms.

Which is the real problem:

Flexible: Yes - Everywhere But the Web

The web side is pretty much, well, krep.  You can smack mouse click into exact locations in a programmatic window that you overlay on an IE region.  And if that sounds like using a laser cannon to heat your Beenie Weenies, well, it is.

AutoIt Final Grade:  D

MacroExpress

Mature and Relatively Flexible

MacroExpress has many of the same powerful windows features of AutoIt but with numerous web features built in.  It is a well supported VB runtime like product, with a relatively active user group and lots of examples. 

It does not handle Java U/I issues, Ajax, Flash, etc.  I'd say that for plain vanilla HTML apps their web automation would work pretty well.  And, yes, I am aware that this is a diminishing crowd.

Well Supported - Not So Much

It costs under $40 and you get about that much support.  The user group/forum seems pretty effective, but there are persistent bug complaints that seem to go unresolved.

MacroExpress Final Grade:  D

iMacro

Very Mature

This product is several years old, is installed in a host of major corporations and startups.   Of all the products, this is most like tools from 'the old days.'   I was reminded more of MultEdit or WinZip or some other product with a cadre of developers and a wide installed base.

Properly Supported

When you buy iMacro (and the developer license starts at $500 and goes up pretty quickly) you get support.  Just like a real product.

Flexible Like A Cirque Contortionist

iMacro can handle Java, Direct Screen, Ajax, etc, etc.  It can even do fuzzy image recognition of bitmapped objects on screen.  Frankly we've been unable to find a situation where we couldn't  bang on an application using iMacro.

iMacro Final Grade:  A

What We Chose

This is probably pretty obvious: iOpus iMacro.  For your amusement, I've placed the candidates on our SEO capability matrix, but I think I can summarize why this really works best: it is the simplest solution.  It has a lot of sophistication under the covers, but a simple glass bottle full of red wine can have a lot of complexity, and history, and artistry too.  So don't be fooled - the buys at iOpus have crafted a specialized tool that eschews the useless and focuses on completing a job just exactly right.

 

Conclusion

We'll start giving some concrete SEO examples using iMacro and some of the architectural framework we've discussed in earlier posts.

Do you look at the source code of Firefox Toolbars that you install?

Few people do. Most people assume that if a toolbar has been written about in the SEO community it must be ok. One such toolbar was making the rounds a few weeks ago. I'm not going to name names, but this article will take look at a toolbar that is actually out there in the wild that you may have downloaded.

Almost every SEO's computer that I've seen has about 30 Firefox extensions loaded. People write articles about how great extensions are, but nobody ever talks about the security ramifications. Nobody ever thinks about what is in their Word Press theme either.

A Very Dangerous Security Hole

A toolbar is an especially dangerous security hole if used for evil. While javascript run from a page is limited by the security functions of your browser, a toolbar runs with the same capabilities as your browser itself. Firefox protects you from cross site scripting attacks, while a toolbar permits it. A toolbar can access your filesystem. For instance, it could save a binary to your machine that you don't know about. You need to be careful about what you're downloading.

Look Inside the Hole

First, how do you go about looking at the source code of a Firefox Toolbar? It's actually quite easy. In Firefox, just right-click on the link to the toolbar and select "Save Page As". That will download a .XPI file to your computer. It's just a zip file, and you can open it with Winzip and take a look.

Is Peeking Allowed?

Isn't it a violation of the Terms of Service to look at the source code of a toolbar? If they've got a TOS that won't let you look at the toolbar code, I'd run for the hills. There's obviously something to hide. Is it protected by trade secret? That .XPI file is a well known format, so it's no more protected than an HTML file you read in your browser. Claiming that's a secret is like claiming a virus is protected by trade secret and you're not allowed to look for it. For the record, this particular toolbar didn't have a TOS. By downloading it, you're basically agreeing to let it do whatever it does.

Real Code, in the Wild

Let's take a look at the source code for this popular toolbar. It starts with something ominious:

eval(function(p,a,c,k,e,d){e=function(c)...

Eval just tells javascript to look at the string inside the parenthesis and evaluate it as if it were straight source code. In this case, our toolbar author has decided to encrypt the source code. If you look in the .js file, you only see a bunch of gibberish, which the code converts to actual souce code and then executes. That's not a good sign. Why would the author shroud their code unless they feared you looking through it?

This type of obfuscation is easy enough to overcome. First, copy the code and run it through a code beautifier. This will output the code in an easily readable form. But it's still encrypted, so we'll just write a little code that looks something like:

function test(p, a, c, k, e, d) { e = function(c) { .... document.write(test('h 6j....'));

This runs the packed/encrypted code through their own function, but instead of evaluating the code inside your browser, merely display it to the output stream. Now you can go looking for anything evil.

What Evil Lurks Here?

Generally, the evil things can be found using the HttpRequest function. That's what tells the browser to send data to another web page, which is how something can "phone home" or do things on your behalf. I didn't find anything especialy evil in this toolbar, but I did find:

this.xmlHttp.open("GET", 'http://api.search.yahoo.com/WebSearchService/V1/webSearch?appid=YahooDemo&output=json&results=100&query=' query '&start=' pos, true);

This code sends a URL to the Yahoo Site Explorer API to find the backlinks. There's nothing wrong with that, except that they're using "YahooDemo" for the application id. That's a violation of Yahoo's TOS. You're supposed to apply for and use your own application id.

The other TOS violation is almost amusing. They're calling Google to get the PageRank of a page, a clear violation of the TOS. Here's the code:

var combination = 16909125; var seed = "Mining PageRank is AGAINST GOOGLE'S TERMS OF SERVICE. Yes, I'm talking to you, scammer."; var doIcare = 'no';

I can't see that the DoICare variable is used anywhere other than as an inside joke by the programmer. He knows he's violating Google's TOS by mining pagerank.

One last thing to note is that this toolbar adds a function to OnPageLoad to check to see if you're on their site. If you are, it does some extra calculations. This is how they add cool stuff to your viewing of their reports. That's all well and goood, but it's also adding the overhead of a function call for every page you ever load again in your browser. Do this with 30 "free" extensions and pretty soon you're starting to slow down.

What, Me Worry?

Why should you care? At some point Google and Yahoo may get upset with your violation of their TOS. You're running the tool from your browser and you're performing the searches that violate their TOS. They may very well just ban your IP. How much of a hassle will it be if you get your IP banned? Was it really worth the time saved by using this toolbar? And wouldn't you rather know what it's doing on your behalf? And frankly, how hard will it be for Google and Yahoo to break this toolbar by changing the name of the demo key or the encryption on PageRank?

There's one last consideration in downloading a toolbar. Firefox has a handy service where a toolbar automatically checks to see if a new version is available, and when you reboot your browser you can automatically install the new version. So how many times do you check for malicious code in a new version of a trusted friend?

If I was a Nigerian scammer or Russian Mafiaso, I'd write a toolbar that did something really cool and get thousands of people to download it. Perhaps even replicate something for free that people currently pay for. The toolbar wouldn't do anything evil. I'd just bide my time while I built up a huge user base. Then I'd make a release of a new version. That version would be a little improvement, but it would also add a function that gets attached to form submission that does a regular expression search for credit card information, then sends that information back to my server. Even if it was spotted quickly (and the chances are that it wouldn't be, because nobody looks at the code of upgrades), I'd still get enough fraudulent credit card numbers to retire.

And if you got caught in that, it would be your own fault. You never looked.

It's a hallowed tradition in technology when you need something: build it. And once built, use it to support your core business. Nowadays big companies call it "Eating Your Own Dogfood." So we know that the Microsofties were tortured with Office/2007 (motto: you thought you knew where things were....) long before any of us.

What do YOU Need to Automate your Social Networking Tasks?

We've looked at minor productivity enhancers and semi-automated tools, but they all lack key elements (automation, reporting) or have architectural issues (RSS Bookmarker's server centric design).

At the end of the day, there is a lack of actionability and ROI. Which comes from solving part of the problem or solving it the wrong way. In the end, what is missing is a technology stack that is focused on solving the end-to-end problem.

Technology Stack

Technology stack is a term of art that discusses the solution's technical elements and their combination, specifically relating each piece of the stack to part of the critical path for the solution of the business problem through technology.

For example, choosing your OS might revolve around current investment, running cost, etc. In this case the choice of Windows/Server over Linux would not only relate to the other technical pieces (ex: VB versus PHP) but to the real world of budget and staff ability.

I believe the minimum technical stack to automate Social Network System usage will look something like this:

Components Included

Client Side:

• Browser (ex: IE, Mozilla, Opera)
• ClientOS (ex: Windows, Linux, MacOS)

Server Side:

• ServerOS
• Database (ex: MySQL, Oracle, etc)
• 3GL "glue" language (ex: Perl, VP, etc)
  
• Reporting (ex: Crystal, Excel)
  
• Workflow and Admin
  
• Web Application Manipulation Language
  

Please note that the server is only conceptually separate from the client. You could certainly run it all on one machine, but if it is built in this fashion you'll be able to bring a horde of minions online when you're swamped with success.

This is the same reason you need to build in your thinking about Admin (user creation, security) and workflow (who does what in which order) from the beginning. You don't have to fully invest in activating it all at first, but if you don't slot it into the design, the retrofit will be horrible. And expensive.

I am certainly not going to get involved in the relative merits of Linux/Windows, Mozilla/IE, language_a/language_b - because we all have opinions and skills there, along with infrastructure and educational investment.

You want to use SQL Server or MySQL or Oracle? Fine, because that means you are solving the database nature of the problem. You want your Workflow/Admin in Nuke or WordPress or Joomla? Fine, because you're solving the problem at the right architectural level with the right tools.

What I want to focus on is the most difficult and arcane piece of the equation, the part that makes this application interestingly different than yet another CRM business application, this piece is the:

Web Application Manipulation Tool

The most basic assumption to start with is that you are developing an application to save you time, money, and to produce a competitive advantage. And that you are not going to write your own web browser with a built in voice recognition and replay capability. In Urdu.

The pointy end of the problem solving spear is a tool that you can use to programatically manipulate a web page: click buttons, insert text from your database, collect results, etc. This is a web application manipulation tool.

It must have the following capabilities:

• Supported application - from a commercially viable vendor or an active OSF-type community
• Mature product - must have developers documentation and have deployment successes
• Flexibile - Must handle a wide variety of web-based applications

Available Options for Web Application Manipulation Tools

We started with a list of over 20 contenders and boiled it down to the the following serious products:

• Macro Express
• iMacro
• CoScriper
• AutoIt

All these products placed strong showings in the above requirements, clearly leaving rivals behind.

To Be Continued.....

The next article will compare and contrast these four candidates and place them on the SEO Capability Grid.

We're trying to debug something that seems relatively simple, but we've got the following bits and pieces in the mix:

• Joomla
• Components galore for Joomla
• PHP
• Ajax
• MySQL
• Oracle
• Two servers (same hosting center)
  
• Development on Windows
• Deployment on Linux
• IE and Mozilla
• Display widgets from DXHTML (awesome stuff!)
  

And some other stuff, I'm sure. Swear to gosh, simple problems can take forever to find.

On the upside, we find that complex problems are easily tackled and that the system's flexibility in meeting the needs of new solutions is outstanding.

True Story

Back in the days when nobody owned a domain and a 24K modem was trick, I was working on an embedded system. We'd compile in a development environment, test, then when it all looked plausible, we'd cross-compile from the x86 environment to the M68K hardware world. (Little Indian to Big Indian for the other geriatrics out there.)

One Friday morning everything stopped working from a software perspective. Everything. Lights didn't go on, lines wouldn't go from low (-5V) to high (+12V) to make the widgets widgetier. Nothing.

We got out the fricking oscilloscope. Nothing made sense.

We dumped the memory (128K of it!) onto some green bar paper and started reverse assembling it back to C. Still nothing made sense.

48 hours later, around 9am Monday, a colleague walked by, asked what was going on, listened to us explaining how we were totally baffulated, and glanced at the much scribbled green-bar.

"Shouldn't there be a memory offset for the pointer to the hardware PROM load right there" he said, pointing at the first four bytes of the printout.

The bug was that we'd somehow forgotten to #include the hardware.h file.

And somehow we'd not noticed that the very first thing on the printout told us the problem.   For two days! 

We fixed the code (10 seconds), cross compiled (5 minues) and were magically right back where we'd been three days ago.

Then we went and got drunk, which was not so easy to do at 10am on a Monday morning in a small Southern town.

Plus Ca Change

The more things change, the more they stay the same.  I betcha we're looking for something really tricky and what we're really seeing is something really simple.

iMacro is an amazing piece of technology and it can help you automate any of your web browser-centric tasks. They have some great examples, a good documentation wiki, and an active support/user community.

Big Savings in Time and Money

We have been big users of this technology for over a year now and have found that it can pay for itself more quickly than an other product we've used. In one recent project we replaced three months of work by 14 people in a data entry center in India ($16,800) with 2 days of programming ($4K) and an enterprise copy of iMacro ($699). We then let six computers chug away for 10 days.

Do the math - time saved and money saved.  The bi-fecta!  What is your time worth?

Review of Technology

We'll do a review of iMacro and why we think it is the technology to pick for automating web based tasks, but for now let's just review the installation.

Download Instructions

Go to iMacro and click on the download link:

 

Then download the iMacros Version 6:

Save the file:

 

Installation Instructions

Go wherever it is you save files when you download them and double-click the "imacros-setup.exe" file. Walk through the normal Windows installation procedure, accepting all the defaults.

When the install is complete you may see a reboot notice. This seems to occur depending on the status of installed Microsoft updates. I tend to NOT reboot at this point but to go onto the next step:

 

 

Make sure you check the, er, checkbox for Clean Install and then click “Start the iMacros Browser.”

You'll see, well, you'll see the iMacro browser. Basically it's a windows program that has the IE browser built in.

Yep, I know what you're thinking: you can do a lot with that. Yes, you can.

Final Step

Now check your desktop for these three icons:

At this point, if you got the ‘reboot’ notice earlier above you can reboot your computer.

Congratulations, your iMacro is fully installed.

Now

If you’re here because you were in the midst of installing some of our tools, well, for gosh sakes, get back where you belong and finish!We'll start posting some handy iMacro code soon, stay tuned.

To be honest, when I started reviewing Semi-Automated Social Networking Automation Software, I expected to find more software like Tag To Rank . Software with an ill defined purpose, with a confusing website full of grammatical errors, and with a distinctly dodgy smell about it.

In general, I didn't find that at all - some fairly fantastic claims and some pretty hard sells, but I've seen worse from Microsoft and Oracle.

Overview

Priced at $297, this software, well, it does something. There are no videos, no examples, but, gosh, I am sure it does something pretty violent with Scuttle since it is a "Scuttle Massive Account Creator (SMAC)."

There are six tabs on the top ("Home", "Support," etc...) and three are broken. The "Product" tab worked and took me to a completely different look-and-feel section selling other, SEO type products.

Again, all without documentation, examples, or demonstration videos.

Summary Grade - Not Ranked

I would avoid spending time evaluating this as a social networking automation tool. Had I not come across the name in, literally dozens of lists of social software tools, I would have moved on immediately and never bothered to bookmark it.