Is your Hadoop cluster big enough to hold your development team’s ego?

“There isn’t a cluster big enough to hold your ego!”

“There isn’t a cluster big enough to hold your ego!”

While Gartner describes a “trough of disillusionment” to describe the hangover that follows a period of commercial hype, on the IT side, I see a corresponding “mountain of ego”. Don’t get me wrong. This is not about a sales guy trying to go after the development community – one I proudly belonged to for many years and where I started my journey in this industry. But ask any developer how long it takes to code something and prepare to be amazed by how fast and easy it all is. Three budget cycles later and a couple of delay notifications and we all know better. Agile development tries to cope with this but it’s no silver bullet.

As companies plough ahead with big data initiatives, the relationship between IT and the business has never been more important.  IT and data integration specialists lead most of today’s big data initiatives; it’s uncharted territory, pioneering work and a place to shine a bright and powerful spotlight on IT’s capabilities and potential to add great value to the business. Challenged by the promise of crafting an algorithm that reads like poetry they dive in head first with the scripting language of choice: Python, Ruby, Pig, Perl, JavaScript…whatever you prefer, too bad there isn’t an Hadoop Assembler library available or we could take some real poetic license!

But here’s the problem. It’s one thing to develop beautiful algorithms and dazzling prototypes but what happens when the inevitable errors, exceptions and irregularities, or – worse – the continuous stream of user changes surface? These unglamorous, seemingly trivial inconveniences that seemed hardly worth factoring into the initial delivery estimates invariably wind up causing major headaches and delays. And fixing them is boring and unworthy of the self proclaimed and highly paid Data Scientist!

The reason I mention all this is that no group suffers the grave consequences of putting ego before pragmatism than IT itself. Business Intelligence 1.0 fell into disrepute for taking ages to implement, costing too much money, being inflexible and plagued with backlogs of IT requests. Our industry must avoid the situation in big data where searching for the Holy Grail of scripting nirvana gets in the way of delivering solutions on time to the business so we all can avoid a repeat failure.

When it comes to data analytics, sidestepping IT is reckless. If you thought ‘Excel Hell’ and ‘rogue spreadsheets’ led to inefficiency and poor decisions, just wait for the mayhem that ensues when ‘rogue analytics’ comes to town! Whether it knows it or not, the business needs IT to handle the data cleansing, warehousing, integration and assimilation that is vital to underpinning fast, meaningful, insightful analytics – especially when big data sets come into the picture.

All this means that IT and the business need to work together and find a rhythm instead of trying to get one over on each other. Since the business rarely gets blamed for bad IT decisions (even when it makes them) this rhythm will only happen when IT gets pragmatic and finds ways to work at the (rapid) pace of the business, especially when using big data sources. In addition to changes in culture and mentality, taking advantage of the promise of big data analytics will certainly involve IT using powerful data integration tools instead of coding scripts and hacking algorithms together. However, through these changes, IT stands to earn respect and even hero status, when the business is able to measure revenue gains and efficiency savings.

On the other hand if IT insists on putting ego before pragmatism, the business will find shortcuts, and believe you me they won’t be pretty! So don’t F* around…visit to learn more.

Davy Nys

*Cartoon drawing by Pentaho’s own Steve Macfarlane

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 12,434 other followers

%d bloggers like this: