Correcting or Avoiding the Data-Rich, Information Poor Problem

Businesses need to plan their information technology infrastructure carefully and align it with business and knowledge goals and objectives at a high level. Once those have been set, IT decision makers must glean from those the precise data that must be collected to achieve them. To do this effectively there must be open communication between the IT department and the departments that will be ultimately the end users. This will help to ensure that there is a true alignment of the business units, business users, and the information technology infrastructure.

A recent trend among business has been to hire analytics, information managers, and DWH personnel. If the firm has an immense amount of data, an employee with these skills is very valuable. In fact, these professionals are in high demand and command very high starting salaries. According to a Business 2.0 article titled “The Hottest Salaries” DBA job salaries are one of the hottest jobs, with salaries in excess of $90,000. It references a job titled “Senior database administrator” ($93,300). Part of the responsibilities of a database administrator is to ensure the integrity of a firm’s data, and to ensure that the data is available at all times.

Another Business 2.0 article titled “The New New Careers” showcases some of the newest jobs that are being created by top firms. Never before heard of jobs are being created like “Information Engineer” to manage and work with the huge amounts of data. Meebo, a new instant messaging start up with 500 million users has created such a position so that they could better utilize the 200 gigabytes of data that they collected each day to make better decisions and to leverage that data in to valuable information. The article also notes that large firms like Pay Pal have also created Information Engineer positions to help compensate for inadequate “off-the-shelf” tools for data analisys. They noted that to really gain value from your data, you must have someone working with, and focusing on the data full time.

Middleware can be a solution to help consolidate a firm’s various information systems and data. For example, through XML feeds, data can be fed in to a central data warehouse at specific intervals in a standardized format, regardless of the underlying technology of the specific transaction processing systems. The data must be scrubbed, validated, and normalized before being stored in the central data warehouse to ensure reliability.

An example of this is the implementation of a drop-ship program by an online retailer, drugstore.com. To integrate thousands of individual merchant’s systems with their own, they contracted with Commerce Hub, which provided the middleware solution. Individual vendors connect to Commerce Hub’s platform, which aggregates all of this information for communication with drugstore.com’s own proprietary systems. This has allowed drugstore.com to greatly increase their selection without major IT infrastructure investments. With all of the drop-ship merchant’s data in one central place, drugstore.com business managers can analyze it to make better business decisions.