work without end [09/18/2008 22:36:41]
I'm just rambling here. You can tell by the way there's no little hand drawn picture on this post.
Which is actually funny, because what I'm rambling about is Quality, which is largely a byproduct of having a Process (love the capital letters) and the real gist of this post is that by sticking to your Process your quality goes up and your headaches go down.
So I've been doing client work, and part of what I'm doing is trying to help fix the broken development process in an otherwise awesome company.
I've been working long hours. I have the dark circles under my eyes. It's 10:47pm right now and there's stuff I ought to be doing for the morning and it's probably not going to get done.
Basically, a product got launched before there was any e-commerce software to back it up. The shopping cart wasn't finished. The code to talk to the fulfillment center hadn't been written. There were no automated tests to be found. I spent several hours the first night manually diffing and merging files into a new CVS repository because there were multiple people working on the site and they didn't even have version control set up.
No... Actually it was worse than that. The company has a version control system but the developers in charge refused to use it.
Anyway, I've been doing a lot of inter-department diplomacy the past couple weeks. Everything's in subversion now, but we're still putting out fires on a daily basis.
I found out today I started one of those fires. I rushed some code out the door because we were in such a hurry and it had to be done right now and so I wrote this really fancy SQL query.
The first time it came back, I found out it was grouped wrong, so I fixed that. Only it was a quick fix because I was overwhelmed with all the other fire-fighting work.
That was a week ago. Today it comes back again, after people have acted on the data, and it turns out there was a massive bug in my SQL:
select if(some_string, use_this, else_this) ...
Do you see the bug? I sure didn't. And I read the if() docs right before I wrote the query.
Turns out that strings are evaluated as false, regardless of their content. (In python and many other languages, the empty string "" is false and all other strings are true)... So basically the "use_this" value never got used.
I'm not ashamed of that mistake. The dang query still looks right to me.
But I am ashamed that I let myself get caught up in the frenzy and shipped mission-critical code with only a cursory visual inspection.
All along, I've been thinking: Let's just get through this launch, and then we can start doing things right.
But after today?
After today, I'm honestly worried that these problems are just going to keep on going. We're so busy fighting fires that we're not doing anything to prevent new ones.
Hell, I'm already worried about the next launch slipping. Everyone seems to think we have plenty of time. But I think we're three weeks late unless we make some massive changes NOW, and this is a project we haven't even started yet.
What's really funny is that on everything else - like the ability to change processes and fix communications problems - I've been scoffed at for being overly optimistic. :)
Anyway, I made the decision today that this has to end. From now on, we write unit tests for every line of production code we create.
That is absolutely the right decision.
And yet I felt like a complete jerk today, telling the guy that just worked overtime 4 days in a row on a poorly spec'd app we absolutely had to have right now that no, we can't ship this after all until we go back and build unit tests.
So this is management, huh?
Actually, I'm having fun.
It's stressful and time consuming, but everyone's been remarkably understanding that it's the process at fault, not individual people. It really is a neat company.
Anyway, I just needed to vent here... I guess I cheered myself up. Optimism comes in handy sometimes. :)
Back to work.