We're trying to debug something that seems relatively simple, but we've got the following bits and pieces in the mix:

• Joomla
• Components galore for Joomla
• PHP
• Ajax
• MySQL
• Oracle
• Two servers (same hosting center)
  
• Development on Windows
• Deployment on Linux
• IE and Mozilla
• Display widgets from DXHTML (awesome stuff!)
  

And some other stuff, I'm sure. Swear to gosh, simple problems can take forever to find.

On the upside, we find that complex problems are easily tackled and that the system's flexibility in meeting the needs of new solutions is outstanding.

True Story

Back in the days when nobody owned a domain and a 24K modem was trick, I was working on an embedded system. We'd compile in a development environment, test, then when it all looked plausible, we'd cross-compile from the x86 environment to the M68K hardware world. (Little Indian to Big Indian for the other geriatrics out there.)

One Friday morning everything stopped working from a software perspective. Everything. Lights didn't go on, lines wouldn't go from low (-5V) to high (+12V) to make the widgets widgetier. Nothing.

We got out the fricking oscilloscope. Nothing made sense.

We dumped the memory (128K of it!) onto some green bar paper and started reverse assembling it back to C. Still nothing made sense.

48 hours later, around 9am Monday, a colleague walked by, asked what was going on, listened to us explaining how we were totally baffulated, and glanced at the much scribbled green-bar.

"Shouldn't there be a memory offset for the pointer to the hardware PROM load right there" he said, pointing at the first four bytes of the printout.

The bug was that we'd somehow forgotten to #include the hardware.h file.

And somehow we'd not noticed that the very first thing on the printout told us the problem.   For two days! 

We fixed the code (10 seconds), cross compiled (5 minues) and were magically right back where we'd been three days ago.

Then we went and got drunk, which was not so easy to do at 10am on a Monday morning in a small Southern town.

Plus Ca Change

The more things change, the more they stay the same.  I betcha we're looking for something really tricky and what we're really seeing is something really simple.