Comment On Slow-Motion Automation

I remember the good old windows API days. AOL proggies anyone?

FIRST ASS~!

"In an effort to save more money, management outsourced the automation project to an offshore team."

You sir, have become your own troll.

Captcha: valetudo (no-one parks a car like udo)

"management agreed to redevelop the PTS and, after a couple of months with a new team, PTS is once again entering data faster than the testers can"

Ah, redeveloping the system -- the solution to all problems. I'm sure PTSv3 solves all the problems of PTSv2 and doesn't introduce any new ones.

FIRST!!!!!!!!!!!1

The Real WTF is that they didn't just use an Access database and have the Excel sheets loaded into there, and the application can use that. What could possibly go wrong?

nikki9696:

The Real WTF ... What could possibly go wrong?

You're mixing and matching your daily dose of IT!

Reminds me of the time where we were writing the new Windows version of a DOS app. A PhD-turned-programmer was given the task of writing the database conversion program, which was essentially a standalone app that read from the old BTrieve version and stored in the new version (yeah, Access).

When the application took 8 hours to convert roughly 1MB of data, I knew something was up. The guy actually defended his program, saying that his program was "carefully moving the data" (I kid you not).

When I looked into his program, I saw that he was actually looping over each record and inserting into Access one row at a time. Worse, he was opening a new database connection each time.

20 minutes later I had the program using bulk inserts. The 1MB converted in a few minutes.

Capsela rocks! I loved those when I was a kid!

(the stock photo for the uninformed)

Whenever PTSv3 needed to find a cell value, it would open the file over the network, read each line out loud through a voice synthesizer into a tape recorder. The tape would then be taken by the office intern to a player on the floor below where monkey butlers would listen for the single value required and etch it into a wooden table. Once completed, the etched table would be mailed an undisclosed location and left near a keyboard where, given enough time, local wild life would learn to copy the words in in return for peanuts.

It was still faster than PTSv2.

Triscopic:

...etch it into a wooden table.

And I thought you were going to take a picture of the wooden table, publish said picture on the internet, and then use a captcha-cracker to decipher the text and insert it into the test form... I guess the wildlife plan works though.

After decades of off-shore programming, I've yet to run across anyone who said, "Yes, they're great!" The height of folly is to continue to do the same thing and expect different results.

You know, maybe it's just me, but I always envision offshore programming teams as a bunch of guys holed up on some old abandoned oil rig somewhere in international coding waters. Or maybe some pirate ship with a good WiFi connection.

Offshore doesn't have to be great, it just has be to good enough. And cheap.

When I was quitting my last employer, they ended up having to switch to guys from overseas-- India and Pakistan-- and so the last two months were basically all working with these guys at night, trying to get them up to speed. The first guy they went with quit out of frustration, as it turned out he "didn't want to learn new things", but the second set of guys took the sketchy prototype I was working on and actually cleaned it up really nicely.

I was shocked-- it turns out that they don't always make things worse. Not that I'd ever want to count on it.

Each successive version of the automation software attempts to reach the goal of "click big green button, sit back, and watch all your work get done in minutes". Doomed to failure by design.

I like the management u-turn as well. That's the real WTF.

Non-standard is for drivers who are rejected due to things like too many speeding tickets, fender benders or DUIs.

Actually, around here that's pretty much standard.

« It didn't take too long for Mike to notice a fundamental flaw in the PTS design -- there was no way to share scripts between states. Although each state had its unique set of requirements, there was a lot of common functionality such as the insured's name, address, phone number, etc. To work around this, they had to simply copy/paste scripts between states and change them as needed. Any changes to the common areas needed to be applied to all 40-plus versions of the script. Though testing time was reduced, programming time quickly skyrocketed. »

If only there was a way to cpreprocess files so as to be able to #include the content of other files; but that would be very complicated without a system to make the files update themselves automagically! Clearly, this is quite the engineering challenge!

My fortune file says: "Those who ignore Unix are bound to reinvent it ... poorly"

Yay, my story got to the front page, and in another column.

PTS2 is still in use, but we're gearing up for PTS3. Yes, I am aware that nothing will be the great "hit button and all work is done" solution, but being able pump a ton of test data through the system still has its advantages, assuming it can be done fast and cheap enough.

The real WTF, not really mentioned in the story, is that the same group did development of the production system. Some of the JSPs are so large dev machines had to be upgraded just to open them. And, of course, the term "code reuse" means "cut and paste" to them, not "put it in a function and call it."

TRWTF is that anybody would actually insure some jackass with DUI's.

amischiefr:

TRWTF is that anybody would actually insure some jackass with DUI's.

TRWTF is that you apparently don't know that many places require you have auto insurance-- so there's a market.

amischiefr:

TRWTF is that anybody would actually insure some jackass with DUI's.

Why not? So long as the cost of insuring him is profitable, there's nothing wrong. Everyone has a risk factor. That's why insurers use that to determine price. Well, in a perfect world. Instead, we have government regulation.

Captcha: eros - Who's horny?

Sean D:

You know, maybe it's just me, but I always envision offshore programming teams as a bunch of guys holed up on some old abandoned oil rig somewhere in international coding waters. Or maybe some pirate ship with a good WiFi connection.

I want the pirate ship job. Where can I send my resume?

Q: What is the difference between PTS and PMS?
A: One bloody letter.



gravis: free gravity

You're too late. The ship was sunk in the last issue of Mandatory Fun Day.

jeffg:

FIRST ASS~!

Let me guess, you used PTSv2 to enter that comment, since it's the second one ... by 4 minutes.

Someone need to learn simple data structures :)

The company I work for tried to outsource the automation of one of our products. The QA staff that had been tasked with automation had been so overworked catching up on the manual testing the manual testers didn't (read 'were not compentent enough to') do, thus outsourcing was a 'good idea'. Despite being warned that the result wouldn't be up to par, hundreds of thousands of dollars were spent. The delivery date which was two months from the start date slipped by more than 6 months. that's right, it was estimated at 2 months and took 8. The result was thousands of scripts, and no common libraries. Loads of duplicated code and a set of tests which didn't test much at all. Tests failed regularly even BEFORE making any changes to the application. And because of the duplicated code the upkeep was impossible. the other products were scheduled to under go the same automation. luckly the effort was scrapped after the first round failed so miserably.

"Because of the data-driven nature of PTSv2, debugging problematic test cases became nearly impossible."

WTF? Why is it nearly impossible to debug test cases because they are data driven?

business analysts could develop the macros instead of programmers

business analysts developing programmers was mistake #1

nixar:

My fortune file says: "Those who ignore Unix are bound to reinvent it ... poorly"

Unix fanboys are bound to praise programs designed for punch-card input as the pinnacle of technology.

Ricky Fine:

After decades of off-shore programming, I've yet to run across anyone who said, "Yes, they're great!" The height of folly is to continue to do the same thing and expect different results.

I've heard some define insanity that way.

On the other hand, if they outsourced the project REALLY HARD, it might just work.

Lars Vargas:

jeffg:

FIRST ASS~!

Let me guess, you used PTSv2 to enter that comment, since it's the second one ... by 4 minutes.

Read it again. It doesn't say "First comment". I think he pretty much nailed it.

The fun thing is that these punch-card input programs mostly out-everything anything else on hardware ranging from supercomputers to clock radios.

operagost:

amischiefr:

TRWTF is that anybody would actually insure some jackass with DUI's.

TRWTF is that you apparently don't know that many places require you have auto insurance-- so there's a market.

Oh no, I completely understand that. What I am suggesting is to stop these fuckers from driving all together.

Merkidemis:

The real WTF, not really mentioned in the story, is that the same group did development of the production system. Some of the JSPs are so large dev machines had to be upgraded just to open them. And, of course, the term "code reuse" means "cut and paste" to them, not "put it in a function and call it."

Sweet, an authoritative "The Real WTF is...".

amischiefr:

operagost:

amischiefr:

TRWTF is that anybody would actually insure some jackass with DUI's.

TRWTF is that you apparently don't know that many places require you have auto insurance-- so there's a market.

Oh no, I completely understand that. What I am suggesting is to stop these fuckers from driving all together.

I'm sure the sort of person who gets a DUI would just quit driving if they couldn't find insurance. Driving without insurance would be illegal!

chrismcb:

"Because of the data-driven nature of PTSv2, debugging problematic test cases became nearly impossible."

WTF? Why is it nearly impossible to debug test cases because they are data driven?

Because then you'd need the guy who writes the test scripts to be able to use a debugger.

Don't laugh. I'm currently sitting opposite exactly this problem at work. Even explaining how to right-click on the test in question and bring up the debugger involved significant investment of effort. Explaining to an "industry veteran of 20 years" (no, he's not outsourced, unless there's a sudden management fad for recruiting zombies) what 'l', 'p', 's' and 'n' might do caused me as much pain as biting off my own foot.

As a general observation gleaned from the last ten years or so, testers seem to like graphs and reports and code-coverage-annotated source and the like. Only the good ones like to analyze. Very few of them are good ones. The result tends to be what you'd get if you hired the underpants gnomes as QA:

1. Preconditions
2. ???
3. Postconditions!

Wow, $1.1 Billion is considered small? Dang, Sounds pretty largeto me!

Jidd
http://www.Ultimate-Anonymity.com

Sean D:

You know, maybe it's just me, but I always envision offshore programming teams as a bunch of guys holed up on some old abandoned oil rig somewhere in international coding waters. Or maybe some pirate ship with a good WiFi connection.

Don't say that. They'll start running Mandatory Fun Day again.

JPhi:

Capsela rocks! I loved those when I was a kid!

(the stock photo for the uninformed)

I wondered if anyone else recognized those. Loved 'em.

pink_fairy:

As a general observation gleaned from the last ten years or so, testers seem to like graphs and reports and code-coverage-annotated source and the like. Only the good ones like to analyze. Very few of them are good ones.

Unfortunately you're right. I've been a professional in the testing field for around 14yrs now and I'm AMAZED how often I come across other testers of allegedly greater experience or training than I have who can't find (let alone analyse) simple bugs.

It seems that current training for 'testers' doesn't include (possibly never did in most places) aspects like software design, analysis and sound experimental techniques.

The situation gets worse when you involve outsourced providers, whether they be application developers or infrastructure providers. They're not only used to dealing with testers who can't really test, but they're expecting it. When you come in and want to do REAL testing, they're sunk because they're hoping that you're going to just sign off so that all the real bugs can be fixed under maintenance charges.

As with business analysis, development, etc., testing effectiveness is mostly derived as a function of the person doing the work rather than industry expectations of the role. I can cut code, design databases, talk to users and developers alike, even though I don't bill myself as a developer, a DBA, a BA, etc. I know enough to develop prototypes, analysis programs, and of course understand the design / architecture / topology of what I'm testing. I don't consider myself a professional in any of these fields. But, I know enough to find more defects than I would with manual processes and work with the development team to refine the application before it hits the streets.

These days I don't really do testing anymore, I'm in a specialist area of scientific data analysis, but drawing on the same skills. Why? Because most people treat testers like crap.

Sure, a lot of them are. But, if that's going to change (and in this day and age of increasing code complexity it HAS to) it's time to start looking at the person, not the role. If you need someone who can analyse to test, hire someone who can analyse. If you recuit zombies, even by accident, ffs don't just throw them into the testing section because you've got nowhere else to put them. Testers won't get better as a collection until we start valuing the role and stop dumping the rugged and buggered into the testing pit because 'they're too dangerous to put elsewhere'.

I'll get off the soapbox now, but just remember - if you have a good tester in your section, treasure them. They'll get you out of tighter spots than you can imagine, because it's their job to imagine those spots in the first place.

And back on topic, the real WTF about this is not that it was difficult to debug using a data driven system, but that after years of manual script development the testing team wasn't up to the task, especially given they had more free time because of the automation in the first place.

amischiefr:

TRWTF is that anybody would actually insure some jackass with DUI's.

You are completely right. When some jackass with a DUI drives someone over, that someone just doesn't deserve to be paid damages. It's their own fault for driving on the same street as previous DUI offenders!

acid:

pink_fairy:

As a general observation gleaned from the last ten years or so, testers seem to like graphs and reports and code-coverage-annotated source and the like. Only the good ones like to analyze. Very few of them are good ones.

Unfortunately you're right. I've been a professional in the testing field for around 14yrs now and I'm AMAZED how often I come across other testers of allegedly greater experience or training than I have who can't find (let alone analyse) simple bugs.

It seems that current training for 'testers' doesn't include (possibly never did in most places) aspects like software design, analysis and sound experimental techniques.

The situation gets worse when you involve outsourced providers, whether they be application developers or infrastructure providers.

<snip/>

And back on topic, the real WTF about this is not that it was difficult to debug using a data driven system, but that after years of manual script development the testing team wasn't up to the task, especially given they had more free time because of the automation in the first place.

Testing may very well be one of those fields where, if you're any good, you shouldn't be doing it in the first place.

What? Well, outside of loonies with an intense preoccupation with oscilloscopes, every single one of the good testers I've come across should actually be coding. Or designing. Or architecting. Or running an investment bank, or something (Chrysler springs to mind).

I think you're confusing "outsourced" with "automated drivel." In fact, the results are very rarely the fault of the outsourced company -- they're cultural. Let me give you a couple of examples:

(1) "We want you to calculate pi to 3,000 decimal places every time somebody hits the big red button."
(1a) <typical bloke in Bangalore> "OK, boss, I've got this PHP code I dragged off the Net that does just that. My boss is happy. You're happy. I know it's cretinous, but then, what am I? A fifth-level software engineer in a hierarchy largely dominated by Brahmins who don't like me talking back."

(2) "We want you to calculate pi to 3,000 decimal places every time somebody hits the big red button."
(2a) <typical bloke in Minsk>"Ah, you think you need the Bailey–Borwein–Plouffe formula. I can do better than that! Let me just check the college rolodex ... yes ... there is man in Dnietropetrovsk who can refine the calculation to yincredible degree! I will code this in Brainf*ck for you! We will win Nobel prize!"

Of course, normal human beings would ask a simple question and make a simple demand:
(1) Why are you fucking around?
(2) Just get it right, damn it!

Unfortunately, the managers who outsource are not normal human beings. Their life is ruled by The Bean. They live by The Bean.

They will probably be reincarnated as The Bean.

Meanwhile, we have to do what we can. Incidentally, I would have accepted version 1 in the OP as the final version and just written a bunch of Perl scripts to munge it for forty different states. But that's just me. Much though I love them, I'm sick and tired of explaining systemic failures to people in Bangalore and Minsk.

Bad programmers are everywhere. You can get lucky finding a good one overseas, but you will often find the bad ones. Exactly the same as when you're looking for programmers in the country. The difference is that you'll probably never meet the guys overseas, probably never understand what they're saying (and vice-versa), hence never actually get to know them and their abilities and hence also don't have a bad feeling when you fire them.

Yay off-shoring!?

amischiefr:

operagost:

amischiefr:

TRWTF is that anybody would actually insure some jackass with DUI's.

TRWTF is that you apparently don't know that many places require you have auto insurance-- so there's a market.

Oh no, I completely understand that. What I am suggesting is to stop these fuckers from driving all together.

That one's easy: make public transit a viable alternative and people will take the bus or the train to the bar and get plowed. Or they'll go to one near their house. As it stands, that's not practical in most places and it's a shame. So the drunks drive cars without insurance.

Downfall:

I'm sure the sort of person who gets a DUI would just quit driving if they couldn't find insurance. Driving without insurance would be illegal!

The sort of person who gets one DUI, sure - the limit is fairly low, and procedure approximates a kangaroo court. Hell, how long ago was it that we heard about the cop who made a habit of writing DUIs for people who hadn't even been drinking?

The sort of person you should worry about is the guy with 3 or 4. But he won't have insurance anyway (or a license). He's got a $500 car wtih a lot of dents in it.

amischiefr:

operagost:

amischiefr:

TRWTF is that anybody would actually insure some jackass with DUI's.

TRWTF is that you apparently don't know that many places require you have auto insurance-- so there's a market.

Oh no, I completely understand that. What I am suggesting is to stop these fuckers from driving all together.

Woaah, you're a trouble-fête. Some fun while shdeering ze wheel should be permitted.

pink_fairy:

chrismcb:

"Because of the data-driven nature of PTSv2, debugging problematic test cases became nearly impossible."

WTF? Why is it nearly impossible to debug test cases because they are data driven?

Because then you'd need the guy who writes the test scripts to be able to use a debugger.

Don't laugh. I'm currently sitting opposite exactly this problem at work. Even explaining how to right-click on the test in question and bring up the debugger involved significant investment of effort. Explaining to an "industry veteran of 20 years" (no, he's not outsourced, unless there's a sudden management fad for recruiting zombies) what 'l', 'p', 's' and 'n' might do caused me as much pain as biting off my own foot.

As a general observation gleaned from the last ten years or so, testers seem to like graphs and reports and code-coverage-annotated source and the like. Only the good ones like to analyze. Very few of them are good ones. The result tends to be what you'd get if you hired the underpants gnomes as QA:

1. Preconditions
2. ???
3. Postconditions!

I also don't know what 'l', 'p', 's' and 'n' might do (maybe print "lpsn" on the screen?). Am I also an industry veteran?

pink_fairy:

...every single one of the good testers I've come across should actually be coding.

Huh? As far as I understand the term, they *are* coding.

Testers are just as much a part of a programming team as the people who type in the C/Java/Perl/Whatever.

The real WTF is when people think that testers are not part of the dev team. That's how we get crappy tests, by making the people who are doing those tests think they are second-class citizens.

B