Note: WTF can stand for “Worse Than Failure”…

Yesterday I was re-living my saga on thedailywtf.com. I’m Jared L.

Part 1
Part 2
Part 3

Well, immediately after reading all of that I went and made a big ‘ole wtf all on my own. Read on…

Every Wednesday and Friday finishline.com gets updated with new promotional material. Most notable of all of these is the “A1 rotator”. This is the flash rotator prominently displayed on the homepage of the website. Each Tuesday and Thursday Sean in marketing will send me (up to) four new .swf files and one .xml file to drop in to the root directory of the site and the A1 rotator will magically update with these new files. I’ve been doing this bi-weekly for a year and a half and have never personally screwed it up. It’s not hard. I copy the files from one directory into another and zip them up and send them over to the build and release manager, Brandon, who sends them over to Verizon who pushes them out to 71 different instances of our website. It functions like clockwork. Until it doesn’t. This is a story about a massive multi-layered failure of the A1 rotator and it’s 100% my fault.

Thursday afternoon at around 4:30 I received the files from marketing and copied them over into the required directory on my local box. As always, I manually uploaded the files to the QA server (a mirror of production) so marketing could test them. I zipped up the files and e-mailed them Brandon. I didn’t bother to go out to our QA server and test them myself because, “hey, I’ve been doing this a year and a half and have never screwed this step up and I’m really busy and what could possibly be wrong? I just copied some files over, what could I have screwed up?” Oh the fallacy…

At this point, the piece of dung sorry excuse for an e-mail client “Outlook” crashed. I had already sent off the files so I didn’t notice.

An hour later when I’m preparing to go home for the day I go to send and e-mail and see that once again Outlook has crashed. I pop it back open and find that I have an e-mail waiting for me from marketing. Five minutes after I sent the zipped files over to Brandon, marketing e-mailed me to let me know that something was wrong with the A1 spot on the QA server. Crap.

I open up Firefox and hit the QA server only to find myself staring at a big empty white space where the A1 rotator should be. WHAT???

Ok, maybe the xml file is pointing at a non-existant swf file and the whole thing is bombing out… What are the names of the files? Oh, there’s a complicated one, “a1_jordan_02102010.swf” maybe the xml has that name screwed up. Let’s simplify things. I rename the file “a1_jordan.swf” and edit the XML “a1_jordan.swf”. Upload. Check. NOPE!

OK, marketing has obviously screwed this all up. Everyone from marketing has gone home at this point so I send an e-mail off requesting corrected files from marketing and prepare to call it a day. But I can’t just leave it at that because that would be irresponsible.

I quickly scan the root directory looking to make sure all the swf files are there. Yes. Is the XML file there? Yes. Ok, so banner_rotator.swf should be pulling them in but it’s crash… OOOOH NOOOOOO (said in a Tim “the Tool Man” Taylor voice). Suddenly I’m flashing back to the previous Thursday when I’m performing this exact same task. I recall that at that time I was feeling a little “cleany” and I decided that I should go through and remove all the old swf files from promos-gone-by. Sean happened to walk into my cube at exactly that moment and together we review the fifty or so swf files that don’t need to be there anymore. Clicking them one by one was going to take too long, so I selected ALL of the swf files and then we went through and de-selected the four required for the next day’s A1 rotator. Once I was sure I wasn’t going to delete the rotator files, I went ahead and deleted the rest of the swf files. Including the banner_rotator.swf master file that pulls in the xml and the four content swfs. SHOOT!

So I recover the banner_rotator.swf, add it to the root, and send it all back over to Brandon. I follow up with an e-mail absolving marketing and I call it a day. Everything’s fine and I even let my boss know what happened assuring him that everything is taken care of. Little do I know…

The next morning as I’m preparing to leave for work I happen to check my work e-mail. The A1 rotator is missing from the site. Apparently the promo push still hasn’t run, and the CDN we use has dropped the banner_rotator.swf file since it no longer exists in our build (because I deleted it last Thursday so it was dropped out of the following Thursday’s production deploy, which was yesterday). Great. The promo push is about to start though and then everything will be ok. I call a few people and send some e-mails to let everyone know what’s going on and leave for work.

When I get into the office I pull up the website and check only to discover that the first spot in the rotation is completely blank. I have several e-mails informing me of this as well. Flash back time… Remember when I renamed the jordan file as part of my debug process? Yeah, me too. Apparently I never put the name back correct but the xml did get switched back. So now I really am missing a swf. I guess the only thing that happens if a swf is missing is that the spot for it in the rotation is just blank. Good thing the only spot that’s blank is the very first spot, which is only seen by 100% of our visitors and only clicked on more than anything else in the entire site!

I call Brandon and sheepishly request that he manually go through all 71 instances of our site and rename the file in every one so that we don’t have to do an emergency push and incur a fine. Which he does and the whole thing is fixed.

At this point I get up and start looking for my coffee mug. It’s missing. I finally stumble across it right next to the coffee machine. Curiously it has a little pile of cream and sugar in it all ready for a cup of jo. It is then that I realize what the root of the whole problem was. I never had my coffee the day before. Why? Because someone took the last of the coffee and didn’t start a new pot brewing; when I went to get my coffee the pot was empty. I had started a new pot brewing, got my mug ready, walked away and never came back; resulting in me doing the entire promo push without any caffeine in me. Apparently I just suck without caffeine. And that boys and girls is why you should always start a new pot if you drink the last of the current one. That is also my lame attempt to shift the blame away from myself, lol. No, this was 100% my fault…