Friday, July 10, 2009

The Great Data Rush.


During the dawn of the 19th century, the world woke up to an era infused with a "free for all" mentality, take what you can and don’t give anything back. Of course I am talking about the era of the now infamous Gold Rush, a time where anyone will become instantly wealthy. Yes, those were the days.

Fast forward 100 years.

Enter the 20th century and all its electronic wizardry, an era of thespians stopping their age to look more marketable and man changing his skin colour and sex. Yes, there have been some weird days on planet earth but apart from those things, we are pretty much still in the 19th century. Confusing? Look at it this way, instead of rare metals, ash and an infinite amount of barrels of dirt. We are now hacking and slashing our way to find a new type of item. What type of item you might ask?

Information, data to be exact.

We live on an age where data is valued extremely high and is considered to be the Holy Grail for most businesses. As they say, what you know might still hurt you but at least it will not unpleasantly surprise you in any way possible. I am not saying that rare metals and other impurities from the ground have been completely replaced by numbers and codes. In fact, that last sentence got my paragraph off the main point which is; Humans have evolved to the point of seeing gold in data and data in gold. In other terms, we learned how to use data to get gold and once in a while, spend some gold to get some data.

That’s where Data Mining comes in. The technology that allows a normal person to see gold encrypted in a wall of numbers and text, the technology that might turn your good business into a great business.


What is Data Mining?

According to the work published by Kurt Thearling, Data Mining is simply the extraction of hidden predictive information from large database.
Data Mining is a vast concept that encompasses a lot
of ideas.

Today’s computers have a lot of processing power, unlike its
forefathers that can only add and divide. With processing power comes great opportunities and it has allowed Data Mining tools to do things that our great- great- great grandparents will surely not believe.

Data Mining tools are expected to predict future trends and market behaviours thus enabling companies make the best possible decision out of many and usually the best decisions are those that are backed by facts. Gone are the days of gut feelings and go with the flow systems.

To mine data you got to have the data and that’s where computer memory comes in. Once again, I refer to the article published by Kurt Thearling, and according to him:

Commercial databases are growing at unprecedented rates. A recent META Group survey of data warehouse projects found that 19% of respondents are beyond the 50 gigabyte level, while 59% expect to be there by second quarter of 1996.


13 years have passed by since that research was conducted and 50 gigabytes of memory in today’s terms are pennies. It is not enough for a personal computer or even an I-Pod with videos and photos. At this point of human technology growth, I expect commercial databases to dish out terabytes of memory, not gigabytes. That is how quickly technology reshaped the world of business.

Data Mining is certainly a powerful piece of technology, so let’s look at what it can bring to the table.

->
Automated prediction of trends and behaviours – Data mining tools would allow the end user to see market trends and shifts in the market at a push of a button. Thus saving time and in the long run, saving money.
->
Automated discovery of previously unknown patterns – Since Data Mining tools can detect trends instantly; it can use the same processing power to detect trends that are stored in its database. Thus giving you more data to analyze and who knows, that piece of data might be your goose that lays the golden egg.

A hammer can be used to pound a nail in a number of angles and with that being said, Data Mining also have different angles of usage. Once again referring to the article of Kurt Thearling, Data Mining have a lot of different techniques for optimal usage and these are some of them:

- Artificial Neural Networks
- Decision Trees
- Genetic Algorithms
- Nearest Neighbour Method
- Rule Indiction

To be able to use these techniques, a programmer or the end user himself would have to develop a Data Mining model that is specific to a certain quandary and of course, don’t forget the system. Management Information Systems can go hand in hand with Data Mining tools. Intranets will make data transition between company divisions faster. Having data about your target market would greatly help because with that you will be able to pick out the trends regarding their product preferences and forecast what products will be strong in the upcoming season.

As a parting note, having the data is not enough to secure success; success is interpreting data the right way. Everyone can read numbers and words but not everyone can see the gold buried along its lines.

8 comments:

  1. "As a parting note, having the data is not enough to secure success; success is interpreting data the right way."

    quoted from the article. This parting note is very fitting. data requires accurate interpretations. One wrong move, and the data could turn against you..

    ReplyDelete
  2. It's a two edged sword. As you said, "that piece of data might be your goose that lays the golden egg." But that same goose might bring you.. bird flu. :P

    ReplyDelete
  3. H5N1 is THE flu.The bird flu. I wanna live.

    ReplyDelete
  4. It's a two edged sword. As you said, "that piece of data might be your goose that lays the golden egg." But that same goose might bring you.. bird flu. :P

    and that could have been applied to almost anything. we also assume that our competition doesnt have the same or better data than us and undermine any decision we take

    ReplyDelete
  5. @Jeff: C6H12O6. 3,7-dihydro-3,7-dimethyl-1H-purine-2,6-dione.

    ReplyDelete
  6. @Barcy: So they say, "never ASSUME, it makes an ASS out of U and ME." :P

    ReplyDelete
  7. @Anna: That's a good one. I've never heard that before.

    ASS U ME

    Haha.

    ReplyDelete