Since I have a low budget for Taiwan,
This Big data competition is an opportunity for me, but it also very challenging!
Just like the word, Big Data = Big size data which content main 4 Vs as shows as below:
Credit: http://www-01.ibm.com/software/data/bigdata/
As software engineer, Our main work is to optimize the 4 Vs when compute the data.
Which mean we have to compute the different kind of large data with high speed and high accuracy.
What make it so challenging is because I just stepped into this field 2 weeks ago, and the algorithm inside is ultimately hard.
Most of them require Artificial Intelligence(AI), and it is my third year course!
Let’s have the most basic example given by the competition organizer——3n+1 problem.
- Select a range of data from 1 to 1,000,000 (Eg: 1 to 100)
- For every number(n), apply the following rule:
- if number is even, divide itself by 2 (n/2)
- if number is odd, multiple by 3 and plus 1 (3n+1)
- every operation(n/2 or 3n+1) mean 1 cycle.
- if the result is not 1, repeat step (a), cycle increase by 1.
- if number is 1, stop and continue next number.
- Between the range, the program have to display the highest cycle for that particular number.
If you don’t understand, can click here for full question.
As programmer, we’ll normally choose a while loop or a for loop and compute every data.
But since it’s a competition, ordinary solution will just kick you out from the list.
After discussing with friend and some on-line study
the better solution is let the program remember the number of cycle which it already compute before.
Mean, the program will become faster(clever) every time it complete a number.
In another word, an AI program.
So does it make a different? Here is the result:
(ignore the wrong answer)
First is a while loop, third is a for loop and last one with memory.
Can you see it such a huge different?
Let scale this up to the real world big data such as Google or FB:
If applying ordinary method,
FB needs 1 minute to load your news feed from database…
With some AI algorithm, the time reduced to 3s for the same operation.
And the beauty of AI is: the more you use, the faster it become!
Now does it make scene for you?
The world’s data is continue growing and expanding.
I believe in the future world, there will be huge demand within this area.
So I really have to start learning now.
However, the example is just an entry question and it already took me a few days to figure it out.
There are still so many things for me to learn.
I hope I can apply AI algorithm in my brain also, the more I learn, the faster I can catch.
Fighting for Exam, Fighting for Big Data, Fighting for Taiwan!
