Oleg Dulin

This week's TDBank debacle and takeaways for IT leaders

2009-10-03T02:55:34-07:00

If you are a customer of TD Bank this past week must have been hell for you. A few months ago TD has aquired Commerce and last weekend they decided to switch over all of the Commerce users to the TD platform. It was a disaster. Direct deposits weren't posting, checks weren't clearings, money couldn't be withdrawn, there were late and NSF fees charged to many TD customers.

My wife and I have a legacy Commerce account and this experience was a wake up call. Commerce was great and we loved it but ever since TD Bank came into the picture the pleasant experience has been destroyed. This debacle was the final straw for us as their customers.

For me as a software engineer, however, there are some interesting points here that I would like to go over. When I worked for JPMorgan Chase I learned that Chase is made up of hundreds of smaller banks that they've aquired over the years. I don't recall reading or hearing about Chase's customers ever experiencing a disaster like this. So, why is it that some banks can go through a merger smoothly and TD Bank couldn't ?

First, let's consider how many banks operate nowadays. Customers use ATM machines, checks, tellers, online banking, debit cards, etc. all throughout the day. Some transactions show up immediately as "pending", others don't show up till next day. Over night the bank runs a batch process that consolidates the ledger files from all of these different systems and business units, reconciliates, and updates customer's transactions. Pending transactions become completed, scheduled bill payments occur, etc.

So, my first question is why do banks still need batch jobs ? In this day and age of instant messaging systems and nearly continuous connectivity, why can't transactions be posted in real time ? I suppose there may be cases where the connectivity is down and customers may freak out that they can't withdraw, but there are ways to ensure fault tolerance and failover. Worst case customers can be told to go to a branch in another town.

Imagine if TD's brokerage unit Ameritrade was 24-48 hours behind at posting equity trades. Chaos would ensue! So, obviously TD has access to real-time transactional technology that the rest of the bank can utilize.

Second, let's consider the basics of a software lifecycle. Most of us in the field would agree you don't just dump a new release onto a couple of million customers over night. According to TD, what happened was they migrated Commerce people over to the TD system and voila the overnight batch process runs twice as long. That was a predictable factor and I don't understand why the processing capacity wasn't increased prior to the release. Did anyone perform load testing ? Why wasn't the roll out done gradually, a handful of customers at a time ?

These are the technical questions I'd like to think about as an engineer. And, as a customer, my wife and I are going to be closing the account at TD over the next few weeks as we unwind the checks and move to a new bank.

Are great developers ever on the market ?

2009-09-07T03:37:14-07:00

I disagree with Joel Spolsky. In his post on why great people are never on the job market he states:

The corollary of that rule—the rule that the great people are never on the market—is that the bad people—the seriously unqualified—are on the market quite a lot. They get fired all the time, because they can’t do their job. Their companies fail—sometimes because any company that would hire them would probably also hire a lot of unqualified programmers, so it all adds up to failure—but sometimes because they actually are so unqualified that they ruined the company. Yep, it happens.

It is true that organizations that fail do so largely because they keep hiring incompetent buffoons. However, calling developers incompetent because they were unfortunate enough to end up in a mismanaged company puts way too much responsibility on the technologists.

A bright developer may get lured into a mismanaged company by an interesting project just to find that project canceled a few months later and the whole team laid off. A bright developer may choose to work with his friends on a startup, then sell the company and move on to other things. He may also choose to work in consulting and go from a gig to a gig every few months.

Joel statement applies more to managers than it does to developers. Great managers are never on the market. They manage their projects well and their companies are profitable and successful. Bad managers don't manage their projects well and end up driving their teams into the ground. Developers are just engineers and that's that. Most of them do what they are required to do to meet a perceived need, and most developers do it well. It is managers that fail.

Friends lists in Java

2009-08-22T07:20:07-07:00

I really wish Java had some sort of access control at the member level that would restrict access by class name, as opposed to just "private", "package-level" and "protected."

I know you can do this with package level access, but with existing code you can't really move things around without breaking it.

Book Review: "Shop Class as Soul Craft" by Matthew B. Crawford

2009-08-09T04:04:09-07:00

When I graduated from college I read "Zen and the Art of Motorcycle Maintenance" by Robert Pirsig. That was back in 2000. Since then I am pretty sure I read it three or four more times as well as Pirsig's other book "Lila." Matthew B. Crawford was a guest on "Colbert Report" promoting his book which I immediately thought was similar to Pirsig's work. In fact, a good chunk of the book cites Pirsig.

You see, I didn't go into software engineering because it is a dynamic field or because I enjoy working with eccentric characters. Not at all. I am a software engineer because all my life the one thing that makes me happy when it comes to work is building something with my hands.

As a kid my favorite toys were erector sets. As I've grown they got replaced with shotwave radio kits and finally in my early teens by a computer. Even though "Shop Class as Soul Craft" is about manual work I find it applicable to software engineering. After all, maintaining a motorcycle requires very similar cognitive skills as maintaining, say, an order management system software.

Crawford makes an assertion some may find outrageous. He questions the value of higher education. We are indoctrinated from early on that we must become knowledge workers. We go to college. We get graduate degrees. Now, consider thousands of MBAs getting entry level administrative jobs that pay far less than, say, entry level computer programmer jobs and you get the point he is getting at. Why, he poses a question, does a plumber charge you $80/hour while a finance major would be lucky to start at $30/hour ? The difference is that a plumber does something tangible while a finance or business administration major knows a whole lot of nothing about nothing at all.

Ever wondered why your knowledge-worker job is becoming dumber and more mind numbing every day ? In software engineer there are hundreds of books written on the subject of division of labor. J2EE Tutorial starts off with the description of technologist roles each one dedicated to their compartmentalised effort. Well, consider this:

Once the cognitive aspects of the job are located in a separate management class, or better yet in a process that, once designed, requires no ongoing judgment or deliberation, skilled workers can be replaced with unskilled workers at a lower rate of pay.

But while the direction of cognitive thought is moving away from the so called knowledge worker, the management is expending a lot of energy on avoiding responsibility for the decision-making aspects of their work:

...one of the principles of contemporary management is to “push details down and pull credit up.” That is, avoid making decisions, because they could damage your career, but then spin cover stories after the fact that interpret positive outcomes to your credit. To this end, upper management deals only with abstractions, not operational details. If things go well: “Finding cross-marketing synergies in the telecommunications and consumer electronics divisions has improved our strategic outlook heading into the fourth quarter.” If things go badly: “Change the Vonage display? That was the kid’s idea. What’s his name, Bapu or something. Jeezus, these immigrants.”

This is not as uncommon as you think. Have you ever emailed your boss asking a question just to end up getting called into his office to be told the answer verbally ? No successful manager wants to commit his decisions to writing. By telling the engineer verbally the blame for failure can be offloaded onto the Dilbert-like character.

I can rephrase the above scenario like this: engineer suggests a performance optimisation; the manager verbally agrees it is a good idea. When the optimisation goes to production and something goes wrong the manager doesn't accept the blame for the decision -- the blame for failure trickles down to the engineer.

Is that the reason many IT managers are resistant to proper software engineering processes that involve issue tracking and release management ? Perhaps. It is definitely an interesting thought to ponder and I will write on it more later.

As software engineers many of us consider working on brand new projects a more glorious and satisfying career. Here is the catch -- brand new projects require far less cognitive ability and can be labor-divided and outsourced. Maintenance projects, on the other hand, require constant dealing with something not of our own making -- much like car mechanics or plumbers jobs. Crawford explains:

Fixing things, whether cars or human bodies, is very different from building things from scratch. The mechanic and the doctor deal with failure every day, even if they are expert, whereas the builder does not. [...]

Because the stochastic arts diagnose and fix things that are variable, complex, and not of our own making, and therefore not fully knowable, they require a certain disposition toward the thing you are trying to fix. This disposition is at once cognitive and moral. Getting it right demands that you be attentive in the way of a conversation rather than assertive in the way of a demonstration .7 I believe the mechanical arts have a special significance for our time because they cultivate not creativity, but the less glamorous virtue of attentiveness.[...]

Any discipline that deals with an authoritative, independent reality requires honesty and humility. I believe this is especially so of the stochastic arts that fix things, such as doctoring and wrenching, in which we are not the makers of the things we tend. Similarly, in art that is representational, the artist holds herself responsible to something not of her making. If we fail to respond appropriately to these authoritative realities, we remain [ed: absent minded] idiots.

Indeed. A prime example from my personal career is learning compiler construction. In college we were taught how to build own compiler from scratch using lex and yacc. I learned a lot more about compiler construction in grad school where I had to modify an existing Ada compiler. Likewise, I find that software engineers who always built trading systems from scratch and never had to support and maintain an existing one are hardly qualified to do their jobs.

Going back to the issue of separation of cognitive thinking from our work, Crawford points out:

We have all had the experience of dealing with a service provider who seems to have been reduced to a script-reading automaton. We have also heard the complaints of employers about not being able to find conscientious workers. Are these two facts perhaps related? There seems to be a vicious circle in which degraded work plays a pedagogical role, forming workers into material that is ill suited for anything but the overdetermined world of careless labor.

Management can't expect workers to be conscientious when their work is reduced to following instructions, processes and procedures. Software engineer can't be conscientious when their work is separated from interaction with business users, sponsors, and analysts. In other words, a good engineer has to eat his own dog food. On the issue of teamwork and office culture, Crawford explains:

we are accustomed to think of the business world as ruled by an amoral bottom-line mentality, but in fact it is impossible to make sense of the office without noticing that it has become a place of moral education, where souls are formed and a particular ideal of what it means to be a good person is urged upon us [...] Where there isn’t anything material being produced, objective standards for job performance are hard to come by. What is a manager to do? He is encouraged to direct his attention to the states of minds of workers, and become a sort of therapist. [...] one finds an imperative for the manager to care, and to sincerely hold forth to his subordinates the possibility of personal transformation. He is not so much a boss as a life coach. The contemporary office requires the development of a self that is ready for teamwork, rooted in shared habits of flexibility rather than strong individual character. [...]

Workers must identify with the corporate culture, and exhibit a high level of “buy-in” to “the mission.” The division between private life and work life is eroded, and accordingly the whole person is at issue in job performance evaluations.

On career vulnerability of managers:

Managers are placed in the middle of an enduring social conflict that once gave rise to street riots but is mostly silent in our times: the antagonism between labor and capital. In this position they are subject to unique hazards. [...] Rather, [ed: their] authority is embodied in the persons with whom one has working relationships up and down the hierarchy [...]

One’s career depends entirely on these personal relationships, in part because the criteria of evaluation are ambiguous. As a result, managers have to spend a good part of the day “managing what other people think of them.” With a sense of being on probation that never ends, managers feel “constantly vulnerable and anxious, acutely aware of the likelihood at any time of an organizational upheaval which could overturn their plans and possibly damage their careers fatally,” as Craig Calhoun writes in his review of Jackall’s book.9 It is a “prospect of more or less arbitrary disaster.” [...]

A good part of the job, then, consists of “a constant interpretation and reinterpretation of events that constructs a reality in which it is difficult to pin blame on anyone, especially oneself,” according to Calhoun. This gives rise to the art of talking in circles. Mutually contradictory statements are made to cohere by sheer forcefulness of presentation, allowing a manager to “stake out a position on every side of an issue. Or one buries what one wants done in a string of vaguely related descriptive sentences that demand textual exegesis.” [...]

When a manager’s success is predicated on the manipulation of language, for the sake of avoiding responsibility, reward and blame come untethered from good faith effort. He may then come to think that those beneath him in the food chain also can’t be held responsible in any but arbitrary ways.

One might be tempted to think this is demoralizing for all involved. But we are highly adaptive creatures, and these circumstances generate their own sort of morality, one in which the fixed points of an internal moral compass must give way to a certain sensitivity and nimbleness. Managers may continue to have strong convictions, but they are obliged to check them at the door, and expect others to do the same. “[M]oral viewpoints threaten others within an organization by making claims on them that might impede their ability to read the drift of social situations.” As a result there is social pressure (one might say a moral demand) not to be too “moralistic.” This pressure is rooted in the insecurity of managerial careers.

And now after such sobering statements I have to take a deep breath. Crawford makes some bold and honest statements about the sad reality of corporate life. Is there much that can be done about it ?

As a software engineer I learned that I find my work far more rewarding when I work closely with the end users. That is, to some extent, why I moved away from working in stratups and boutique consultancies and went to work directly for the customer. I am the happiest when I work with the customer and discover a problem to solve, and then I am given an opportunity to deliver and succeed.

Crawford describes his experience of getting paid for motorcycle repair -- where a wad of $1000 in cash given to him by a happy client was somehow more rewarding than the same amount given for an office job. The reason is because the motorcycle either runs or it doesn't, while at the office he was not being judged based on the results of his work.

First time I got paid for building a piece of software was when I was 13 and I wrote a Lotus 1-2-3 macro for my mom's business partner and got paid $50 for it. That is not a lot now but for mid-90's Ukraine it was. When I worked for a couple of years as a consultant getting my timesheet signed every week and realising that my customer will be utilising my services for another quarter was highly rewarding. Likewise, I am uncomfortable in environments where metrics for performance and compensation are arbitrary and subjective, as most engineers would be.

In any case, whether you agree with Crawford's views on corporate culture or not he does offer a sobering and honest viewpoint on the modern work ethic. It is a must read for anyone considering themselves an engineer and even a manager. It is one of those rare books that offer a society a look in a mirror undistorted by modern culture.

Shop Class as Soulcraft: An Inquiry Into the Value of Work

Re-educating myself

2009-07-16T02:43:38-07:00

I signed up to take a few courses at NYU towards a certificate in "Financial Risk Management." This fall I will be taking an "Asset Securitization" course.

When I was doing my Masters at NYU I could've taken a few electives at the school of business but I chose not to. I was preoccupied with learning application server design and formal software engineering methodologies. At the time it was useful. In retrospect, though, I wish I had more business training.

So, that is why I swallowed my pride and decided to go back to NYU for some continuing education courses. I wanted to select a course of study that I would find interesting as an individual, something that would be valuable to my current employer and something that I could leverage in the future. "Risk Management" fit the description.

This makes Eclipse Galileo run faster

2009-07-06T08:35:35-07:00

I observed that the new version of Eclipse appears to occasionally underperform (to put it lightly) and just sit there consuming 25% of the CPU cycles. The following Eclipse startup parameters appear to have helped:

-vmargs -Xverify:none -XX:+UseParallelGC -XX:PermSize=20M -XX:MaxNewSize=32M -XX:NewSize=32M -Xmx256m -Xms256m

Eclipse.org home.

What IT Leaders Can Learn From Google's Staffing Problems | The Accidental IT Leader

2009-07-06T05:39:17-07:00

Here is another interesting article from "The Accidental IT Leader" -- Google is having staffing problems ? Is this an opportunity or an ominous sign ?

What IT Leaders Can Learn From Google's Staffing Problems | The Accidental IT Leader.

If you could go to work for any company out there right now, which one would it be? A lot of us would say Google – everything that we’ve read and heard about the company makes it seem like a great place to work. However, it turns out that even Google is not immune to IT staff problems

The Accidental IT Leader: How Can A Manager Manage IT Workers When There Is No Company Loyalty?

2009-07-04T10:25:34-07:00

"The Accidental IT Leader" writes:

Welcome to the 21st Century where all IT workers now view themselves as temporary workers. The constant cycles of downsizing and outsourcing have made even the most committed workers view their jobs as being not so much as a career, but rather as a temporary pit-stop.

I can't help but think about the underlying theme of my career since graduating from college in 2000:

In 2000 I felt a full-time job at IBM Research was pretty much guaranteed to me. I worked my way through college at IBM and had a great relationship with my management there. So, even my boss at IBM was disappointed when he found out he can't hire me after all because IBM froze hiring in the research division as the dot-com bubble burst.
The manager at a pure-play consulting firm that hired me quit after I was there for about a month.
The dot-com I went to work for afterwards went out of business within 6 months of my starting there in 2001.
I had more managers than the number of years I worked.
I had the best sense of my career path when I worked, ironically, as an independent consultant on an hourly basis. I felt in charge of my career -- the better the impression my customer had of me, the greater my chances were of landing more business with them or their contacts.

With this experience do I have a reason to believe in company loyalty ? I don't think so. I don't go to company end-of-the-year parties, I avoid making friendships with other employees. I do maintain a network outside of my company and I keep up with my skills. I leave the most minimum set of personal items at the office. When I work I do my best but I try not to expect any career development. Perhaps, I need some time to get out of this holding pattern -- but it is not easy.

Links:

The "EMACS" room on a cruise ship ?

2009-06-30T12:12:43-07:00

Look at the picture of "Camp Carnival" and click "View Larger". It says "EMACS" on the wall. I am not aware of any definition of the word EMACS other than the best text editor ever.

Carnival Miracle Cruise Ship, Carnival Miracle Reviews - Carnival Cruise Lines.

Happy father's day

2009-06-21T09:35:15-07:00

I am having a flu-infested father's day this year. But it is the events like this that make me realise what is important in my life.

My 2.5 year old daughter goes to a daycare. Surprisingly, she didn't get sick since February. With nothing else to worry about I started worrying about my career development again. That was until last week when she caught a flu.

Lucky for us, my wife works in a public school and the school year is over. So I had to stay home with our daughter for a few days and then next week my wife can take over. I took our little girl to the doctor, took care of her, and of course as I anticipated this weekend I came down with a flu.

My daughter is being really cute about it. Yesterday, she covered me with a blanket on the couch, brought me a pillow and TV remote. She gave me a kiss on the forehead. While my wife was out getting me Tamiflu I couldn't entertain our toddler -- so I offered her to watch 3 cartoons instead of her usual two 5-minute Scholastic cartoons. She said, "That's ok, daddy, if it makes you feel better I will only watch two" followed by "Can you please get better so you can play with me?"

I feel a bit better today. Tamiflu is kicking in and the fever is not as high anymore. I still feel like crap, but my family taking care of me is the best father's day present I could get.

Links for 2009-06-10 [del.icio.us]

2009-06-11T00:00:00-07:00

Links for 2009-06-09 [del.icio.us]

2009-06-10T00:00:00-07:00

Java play WAV sound file - AnyExample.com
"There is source code of simple class AePlayWave in this article, which can play WAV(AUFF, SND, AU might also be supported) sound files asynchronously (in a separate thread, without interruption of main program). It is possible to use it in console or GUI Java programs for playing user notification sounds. There is also trivial RSS-feed checker, which plays various sounds on rss-feed update. '

Links for 2009-05-26 [del.icio.us]

2009-05-27T00:00:00-07:00

Apache Directory - LDAP and Kerberos, entirely in Java
"The Apache Directory Project provides directory solutions entirely written in Java. These include a directory server, which has been certified as LDAP v3 compliant by the Open Group (Apache Directory Server), and Eclipse-based directory tools (Apache Directory Studio)."

Links for 2009-05-22 [del.icio.us]

2009-05-23T00:00:00-07:00

LinkedIn: Answers: What's after high school? or college?
"If you are in a position to give guidance or guidelines to young people who will soon graduate from high school, what would that be? For young people graduating from college, would your guidance, advice or guidelines be of any difference? "
JCS - Java Caching System
"JCS is a distributed caching system written in java. It is intended to speed up applications by providing a means to manage cached data of various dynamic natures. Like any caching system, JCS is most useful for high read, low put applications. Latency times drop sharply and bottlenecks move away from the database in an effectively cached system."

Links for 2009-05-21 [del.icio.us]

2009-05-22T00:00:00-07:00

Globalization and Offshoring of Software
A report of the ACM Job Migration Task Force

Links for 2009-05-15 [del.icio.us]

2009-05-16T00:00:00-07:00

PartnerUp - Redefining the way Business are built -
Make Meaningful Connections with Other Small Businesses and Entrepreneurs

Links for 2009-05-06 [del.icio.us]

2009-05-07T00:00:00-07:00

Java EE Tutorials