TDWI Webinar on in-memory data, another miss

I’ve been skipping the last few TDWI webinars, not exactly knowing how to politely criticize some poor ones. However, I feel as I’m performing a disservice to those who read, so I’ll have to discuss today’s.

The title was “In-Memory Computing: Expanding the Platform Horizon Beyond the Database.” The pitch was that in-memory is so good for databases we should think about doing everything from ETL to everything else in the information chain there. One word critique: Oy.

In-memory analytics has been great for very fast processing. Having the data resident in memory is obviously a great way of providing rapid response for users of reports and analytic tools. However, it’s no panacea.

Simply put, two demands are limiting the cost effectiveness and even ability to do in-memory analytics: The amount of data and the number of users.

One of the repeated refrain of in-memory proponents is “memory is cheap!” Yes it is. However, massively parallel servers with the ability to efficiently link multiple cpus to large amounts of memory while providing coherence for multiple users aren’t. They quickly get very expensive with the costs of high end machines being much more on a pure memory amount level than commodity servers. There’s also an upper bound and with much of the larger data analytics today, multiple servers will be needed.

The other issue is the growth of self-service BI and mobile access to reports means that more memory is needed for non-database usage. A number of in-memory solution providers tell you that each user takes space in memory to satisfy individual needs. The more users the more space is taken from database availability.

The growth of server farms, being created in the Cloud now, is how the blend of in-memory versus space requirements will be addressed. “Fast enough” matters more than millisecond response time. With what we are constantly learning about both data manipulation and presentation, the strongest Cloud providers will win by keeping the most used information in-memory and sharing the rest among caches on multiple servers.

In-memory isn’t new and needs are much different than when it was. Listen to people talk about it and pay attention: If it’s the only thing discussed, they’re not being honest; if it’s a core part of the solution with its caveats addressed, the vendor, analyst or pundit is helping you.

Leave a Reply

Your email address will not be published. Required fields are marked *