Monday, October 1, 2012

Don't drink the Methodology of the Day Kool-Aid

I work with some of the largest amount of data on Earth

In the Enterprise world, it's no doubt that Java has been very successful. Java has invaded corporations like there's no tomorrow, promising reusable code, developer support and frameworks that make web development for the enterprise one of the best of its class.

A little bit of Background

Before joining a FORTUNE 50 company, I was the biggest adopter of the "Development-Driven-Development" methodology. (Don't worry. That methodology doesn't actually exist), but it gets my point across perfectly. I left Wolfram Research as a Software Engineering Manager. My team was used to creating and delivering products very fast. We were doing so good that I was promoted twice in the same year and 5 times in 4 years. It was a great feeling of getting things accomplished and being able to move on to the next target.

My team was creating something beautiful. It all started with a series of Import and Export plugins for Mathematica. After a while, the team expanded to Import and Export, Data Curation, and the Image Processing team which I managed. Together, these teams basically created what is Wolfram Alpha today (the backend part at least). We went from being able to import GIF and a few other raster images to a steam powered machine that could import almost any format in almost any field such as 3D models, spreadsheets, Bioinformatics, etc. We were also able to "Rasterize" any type of Mathematica expression into a Raster image which is still the method of generating images in Wolfram Alpha today.

Why am I telling you all of this? Because coming from a fast-paced environment where new things happened everyday to a FORTUNE 50 company was like night and day for me. Things like innovation, creativity and motivation were replaced by process, methodologies and frameworks.

The reality of a corporation

A Corporation's IT sector cares deeply about HOW to do things. They are not so much focused on being cutting edge as they are in creating ways that can be easily reproduced and how to provide support for these solutions in the long run. It looks for cookie-cutter software implementations that can be executed and maintained by ANYONE even when the paradigm doesn't apply. They believe in this way of doing things so much that they are willing to adopt almost any kind of framework or methodology that promises to create a process around a solution. What I mean by that is that once a vendor gets in the door, they will be able to sell almost anything else. All they have to do is throw in a few buzzwords and it's adopted. That's the goldmine of the IT industry: Being a preferred vendor for a large corporation.

Back to the subject...

Suddenly I went from creating cutting edge applications to having meetings about which framework was in fashion that day or how we could use the newest Spring Loaded Inversion of Control Toolkit. Applications became so bloated that something that I could write in one week by myself at home would take 10 people more than 1 year to create. People were obsessed with the buzzwords and were willing to do anything to start using the newest technology just so that they didn't have to write real code.

I was getting desperate. I couldn't agree with 99.9% of what my leaders were saying and I had no way to fight against them. In the old world I could tell my VP or CEO my idea was better than theirs and they would listen to me (as long as I could back what I was saying), as for the new world, technical expertise doesn't matter if you don't fit the mold. You may be the brightest Engineer in the room, but when someone with a higher pay-grade says something, it's written in stone and can't be changed. It's a world were admitting you are wrong or somebody has a better idea than you is a sign of weakness and a sign of weakness prevents you from climbing higher in the management ladder. I can name several occasions where a superior product was questioned and vetoed because it wasn't on the approved software list / framework / language. Some times I think this is actually the perfect plot where the goal of a vendor is to make a solution as complex as possible so they can keep sucking the blood of the company for years to come.

The best example I've been part of is a project where a C# application, which was originally written as a single class, needed to be converted to a Java / WebSphere application. After a whole year of coding and 4 developers working full time, the application required 15 JVMs on 15 different servers across the country and things ran as slow as a one-legged turtle. The great thing about it was that it had the latest technology, Dependency Injection for everything, and functions written exclusively for OR comparisons. Testing this application was a nightmare. Testers needed to get 7 levels deep in the code just to see how something worked. Eventually once people realized what had happened, it was too late to turn back.

This is about the time I learned about the Spring Framework and Injection. In theory it might be an awesome idea, but Dependency Injection is not useful in practice, trust me. Dependency Injection makes an application so complicated with all the XML configuration files all over the place that the time it might save you from recompiling your app when you want to change something will be multiplied by 1000x when servicing this same application. Of course I speak from personal experience, but I've lived on the cutting edge and now I live in the conservative world. I know very well what produces results and what costs money.

Time and time again I see that developing in an environment like this is like running a marathon with your legs tied. While the latest buzzword promises to make applications easier to develop, they will make developers cry when they have to maintain this type of code and I guarantee that it will take at least 3 times more resources.

To summarize

Don't take the new buzz words out there as the holy grail of development, be it Test-Driven-Development, Design-Driven-Development, Dependency Injection, Inversion of Control, etc. Analyse what your application needs. Talk to your customers. Think of how to service it and think of what you are trying to achieve. Everything has a place, but while knowing these things might make your resume look nice, they may not be the best way to solve the problem at hand.