View Sidebar

Post Tagged with: Big Data

Hadoop Can Come Handy Even When You are Not Dealing with Big Data

Hadoop Can Come Handy Even When You are Not Dealing with Big Data

Hadoop was developed to cater to the needs of web and media companies for managing big data. But even if you don’t have to deal with big data, you can still use Hadoop in many ways to enhance your data and resource management. Today Hadoop is being used by almost every business, whether they have big data or small, to manage their data.

The Main Features of Hadoop

The main feature of Hadoop is the HDFS storage system. HDFS stands for Hadoop Distributed File System that operates on low cost hardware.

MapReduce was developed for resource management and data processing but with Hadoop 2.0 it has been left just to focus on data processing while YARN is used for resource management.

These features of Hadoop can be utilized in many innovative ways by big and small businesses.

Data Archive

One straightforward use of Hadoop is to archive data files. Since HDFS runs on commodity hardware it is simple and cheap to scale so businesses can start small and expand as their business grows. They can store all their data at a very low cost.

Instead of destroying data after the regulatory period is over, companies can store decades of data and analyze it in real time to help their decision making process.

Data Staging Area

Traditionally ETL tools are used for extracting and transforming data. When Hadoop came to the scene, it could have killed ETL forever if ETL providers hadn’t been smart enough to provide HDFS connectors so that Hadoop could be used along with their ETL software.

By using Hadoop you can store the application data and the transformed data in the same place. This makes it easier to process the data at a later time and reduces the time to process the data. Hadoop can help ETL in improving data processing.

Data Processing

Instead of sending data to the warehouse and then use costly resources to update it in the warehouse, you can use Hadoop and its MapReduce function to process and update it before it goes to the warehouse. Hadoop’s low cost processing power can be used not just for your warehouse data but for other operational and analytical systems as well.

HadoopHadoop is a very powerful tool that can help all businesses to handle their data in a better way. You don’t have to be sitting on top of big data to use Hadoop. You can start even when you have small data and Hadoop will let you collect decades of data till it becomes big data and then you can start making use of all this data by using big data analytics.

IT Companies Should NOT Hesitate to Milk Big Data

IT Companies Should NOT Hesitate to Milk Big Data

Big Data is being utilized by businesses in every field to answer questions and predict the future with more reliability than the traditional methods that were used before big data analytics came along. It is surprising that IT isn’t utilizing big data as much as they should.

IT, like most other businesses, needs to predict the future in terms of surprise requirements, new opportunities and threats and worst case scenarios. A lot of these questions can be answered by using big data analysis. All parts of IT; operations, security, customer service, forecasting etc. can benefit by using big data.

IT has access to a lot of data in terms of logs, traces, emails, counters, feedback, polls etc. that it can use to solve critical problems related to predicting future scenarios.

Big data

The Usual Solution

The solution that IT has been using till now is to purchase packaged applications for their services and in case of a unique solution, they tend to integrate their own solutions to the applications with the help of their unique business know how.

The new idea is to use all the data collected, both internally and externally and apply big data analysis to it to move on to the next level.

It’s Already Started

A few IT companies have already started utilizing big data because they see the potential behind it. EMC IT is using big data to analyze their data and predict potential issues with their app delivery system.

Some companies are starting to use the huge amount of data they have collected for security. Already certain applications exist that analyze security data but by creating their own applications companies can come up with better solutions for their unique security requirement.

Companies can also use big data to forecast how much money they’d need to spend in the coming year on upgrading their capacity to store, compute and analyze data.

IT Always Leads the Way

It has been seen that the IT is always the one that takes on new technologies and systems and gains enough domain expertise in the area to help other branches of business later.

The data already exists and the tools for analysis of this data also exist. It’s just a matter of time before all IT companies start using big data in new and creative ways to solve unseen problems and predict the future. It is easy to get started and a lot of money is not required to get into big data. We are going to see a game changing utilization of data in the near future, just as soon as the IT sector wakes up and smells the data.

Apple’s Acquisition of Topsy Could be a Great Move

Apple’s Acquisition of Topsy Could be a Great Move

Apple has recently acquired Topsy Labs, a San Francisco based analytics firm, for $200 million. The Indian owned firm specializes in analyzing tweets and has an archive of over 400 billion tweets, starting from 2006 when Twitter was first launched. It also has a searchable database of all tweets ever sent, something that even Google doesn’t have, because of its partnership with Twitter.

topsyTopsy Labs was formed in 2007 by Vipul Ved Prakash, Rishab Aiyer Ghosh, Gary Iwatani and Justin Foutts. It collects and analyzes huge amount of data generated everyday on Twitter and allows its customers to make sense of all the noise. This buy could be a great move for Apple depending on how they use Topsy.

Apple hasn’t commented on what they intend to use Topsy for, but it is clear that with such a powerful tool, they have a lot of opportunities in front of them. Kristin Huguet, a spokesperson for Apple Inc, confirmed the acquisition but did not reveal Apple’s plans for Topsy.

“Apple buys smaller technology companies from time to time, and we generally do not discuss our purpose or plans,” Huguet said.

The Opportunities for Apple

The most obvious use is to get feedback about their products and services and market their products in a better way. They could also analyze key players on social networks and use them to spread the word about their services.

dataTopsy has developed many state of the art systems to deal with such huge amount of data obtained from social networks. They have filed patents for many systems and methods for customizing filtering and predictive crawling of social media. Apple could use this technology to analyze the data that they have been collecting through their devices and app store.

They could use it to improve the search feature of Siri, providing users with better search results by analyzing what recommendations their friends made on Twitter. Twitter is the most used social media platform on smart phones and getting access to all of its data along with the ability to analyse it, is definitely a great opportunity.

Competing with Google

Google has been trying to make their search engine more personalized based on data gathered from social media. Now Apple has a very strong tool to do the same. Both companies are spending big money on acquisitions in order to maintain their edge in the rapidly changing world of information technology. Apple spent $496 million in their last fiscal year on acquisitions while Google spent a whopping $1.4 billion.

Is Big Data a Threat to Your Privacy?

Is Big Data a Threat to Your Privacy?

Big Data is growing bigger every day and along with it the concern over invasion of privacy is also growing. Tracking all the data generated by your mobile and other devices and your interactions on social media, is beneficial for advertisers to tailor their ads to suit you. But there’s more to the story than that. Companies have now begun to come up with very creative ways to use real time data.

Let’s look at some interesting examples.

Smart Rubbish Bins in London

An advertising firm in London came up with the idea to use strategically placed dustbins to track the wifi signal of phones of the people passing by. They could use the serial number of the phones to track the movement of every individual. They could then use this data to show advertisements on the screen of these bins, that are targeted at the person passing by.


Now even dustbins are becoming smart!

The officials have asked Renew, the responsible ad firm, to take down the smart dustbins as there has been a lot of concern about the invasion of privacy of the people.

Police Cars in Australia get Number Plate Recognition Cameras

The Aussies have come up with another great use of Big Data by using number plate recognition cameras that can read multiple number plates simultaneously and also search their database to find out all the information about that driver. They can tell if a car is stolen or if you have unpaid parking tickets just by looking at your car’s number plate.

police car

The hand of the law gets longer.

Are Such Examples a Threat to Your Privacy?

When CCTV cameras first came on the scene, the public responded to them with an outrage similar to what we see now in terms of Big Data. But once people got used to the new technology and saw the benefits in solving crimes and catching miscreants swiftly, the fears of Big Brother always watching them subsided.

The truth is that people will allow collection of any data as long as it is collected with their permission and it is used to create value for them. Instead of shoving ads in people’s faces, companies should try to find other ways to use Big Data, not only to reduce costs for the company but also to provide quality to the customer.

One great example to highlight the creative use of Big Data is the potential for insurance companies. Today all natural or man made calamities generate a lot of data in the social media.


Data about Hurricane Sandy

Insurance companies can use this data along with before and after images on Google Maps Street View, Flickr, Instagram etc. to find out how much destruction of property their clients have suffered.

torn houseThey can estimate the number and amount of claims that they will have to deal with. They can provide quick claim settlements to their customers which will be appreciated by all and people will readily agree to data collection if they are told of such rewards.

Great Opportunities

A Westpac survey showed that it only took 30 months for mobile usage to reach 1 Million as compared to 80 months it took for online usage to reach the 1 Million mark.

graphThis means that there are great opportunities available to use this rapidly growing Big Data but it will have to be done with care and while keeping the interests of the consumer in mind.

Apps may minimize need for big data scientists

Apps may minimize need for big data scientists

If technology can be backed, then soon we could have a new breed of applications that would allow businesses to create their own big data apps in cloud, thus minimizing the need for IT and big data scientists. 

big data appsBig data, a word that was brushed away as a buzz word by many technology scientists has made it presence felt in a big way. Even the scientist that dared to believe it as only a buzz world have acknowledged the importance of big data for enterprises.

Considering the hype around big data, recently a survey was conducting by SAS institute. The survey concluded that there is a significant rise in demand for qualified data scientists, who can find decisive results from the avalanche of data.

The report revealed that there is an expected 243% increase in demand for big data specialists in UK alone in the next five years.

The opportunity for big data scientists is enormous, since there isn’t enough number of experts on the ground. But is this demand only hype on the big data expansion, or is it time for universities and institutes to churn out armies of qualified big data experts?

According to an article by Successful Workplace co-founder and marketing executive, Chris Taylor, on Wired, there are good enough scientists around already, and as more institutes and courses spring up, producing data analysts will be an automated effect.

An infographic – world needs more data scientists, points out that ‘data scientist job postings increased 15,000 percent between 2011 and 2012 alone’. And people seeking career in big data technologies could soon become the most sought-after people in the IT industry.

However, according to Chris Tylor, the word big data scientist has just been coined a few years back and thus there is so much demand for it. He further that numbers don’t really add up, he is off the view that data scientists have always been there in the IT industry, it just the role is more demanding this time.

According to Taylor and a point that really strikes us is that, given the exponential growth of technology, data scientists could soon find themselves replaced with new applications devised to simplify the process of big data analytics.

New apps could be launched soon, which will allow businesses, governments and individuals to create their own big data apps in the cloud with much lower number of data scientists. Citing this Taylor believes enterprises should refrain from rushing into hiring too many data scientists for now.

Via: SiliconAngle

Cloud computing and big data enable real-time decisions

Cloud computing and big data enable real-time decisions

Big data analytics and cloud computing together have a potential to act as a huge springboard to real-time decision making. As big data and cloud become increasingly intervened, it is leading to more efficient and accurate results in real-time. 

A Close relationship between big data and cloud computing coupled with real-time processing capabilities will give birth to data analytics proficient in producing real-time results that can change the way companies build and market products. This sentiment was recently expressed by Amazon Chief Technology Officer, Werner Vogels while talking to The Guardian.

Cloud and BigData

Tools and technologies of big data offer new automated ways to condense large amounts of data into understandable format instantly. Unlike the conventional business intelligence, which had limitation of being futuristic, big data has the ability to not just analyze data about what has happened, but also has the potential to process currently produced data in real-time.

In Vogels’ view, big data and cloud computing share a very close relationship, since it requires no limits to store and compute. And when this ability is combined with real-time processing, data analytics will rise instantly to produce real-time results for companies using it. This will allow the enterprises to take real-time decisions based on real-time processing and analysis of data – business will be able to make decisions based on current status and not on the past information that was the case until.

Imagine a scenario where companies could take manufacturing and marketing decisions based on the most current data, which is stored in cloud and analyzed by big data tools. How effective will be the decisions that’ll be taken on the information available right now?

This is the kind of radical impact business decision making could experience if they can figure out ways to use cloud computing and big data in harmony.

It is evident that cloud and big data make a good team, considering the fact that you can store as much data as possible on cloud, as and when it comes. But there is a point where businesses will have to consider when this collaboration is becoming too expensive for them.

Korea opens first big data analytics centre

Korea opens first big data analytics centre

To bring its technology sector at par with the global technology giants, South Korea has set up country’s first big data analytics center. The big data analytics centre will enable researchers and businesses to refine and analyze big data for their projects.

South Korea’s ministry of Science, ICT and Future Planning have worked in collaboration with National Information Society Agency (NIA) to open ‘Korea Big Data Centre’ (KBiG), the country’s first big data analytics centre in the NIA building.

Big_Data_center south koreaThe Ministry of Science informed that the Korea Big Data Centre is set up to promote data analytics in government and private sector with the intention to transform Korea into a data driven creative economy.

The new centre will allow all kinds of researchers, businesses and hospitals etc., to process and analyze the avalanche of data for their products.

The aim of the center is to bring the Korean industry at par with the global technology giants like Google and Amazon, who have gone ahead by 3-5 years by properly enrooting their big data.

Korea Big Data Centre is primarily created for small and medium size businesses, universities and citizens who wish to analyze their big data to find answers to their research or business issues.

Until the opening of KBiG, universities, research institutes and businesses in Korea were paralyzed when it came to securing and analyzing big data. They had to rely on limited IT infrastructure, they could create by themselves. Korea Big Data Centre will now provided the much needed shared service that everyone will benefit from.

According to an official from the Ministry of Science, the new centre is expected to be a “test bed” for big data analytics to foster research at universities. This, he says, is a basic solution for everyone to use the services in order to analyze big data.

Via: FutureGov

11/27/20131 commentRead More
3 ways big data analytics is changing our lives

3 ways big data analytics is changing our lives

Big data has travelled a great deal from just being a buzzword only a year back. Service providers and analytics are fully engrossed in understanding the benefits which can be derived out of big data analytics. Since big data is everything we do every day to leave a digital trace, it can be analyzed to better our lives. Read on to know how big data analytics is changing our lives with each passing day.

big data changing livesWith the explosion of data in every form, from books to maps and from calls to apps, from advertisements to social network updates and from surveys to varying trends – we are leaving more and more digital traces with every digitalizing world. Since all this avalanche of data is touching our lives every day, it has become one of the mega trends that will impact (and is impacting) everyone in one way or the other.

Listed below are a few examples of how big data analytics is already changing our lives for good.

At home and office

There was a time when nothing was possible at home without human intervention, today we have small and seasoned manufactures developing devices that help us monitor every single thing at home, from elders to pets and from appliances to door locks, everything can be monitored from miles ways just by using a smartphone and an application.

Manufacturers have devised sensors to automate lighting and your appliances at home or in offices. You can turn on the lights or the air conditioner even before you walk into the office or home. You don’t have to stand up and turn of the light or walk to the washing machine to switch it on, a smartphone with dedicated app can do it for you from the comfort of your bed.

While driving or shopping

Auto manufacturers like Ford are using big data analytics to make their future vehicles more environmental friendly. In addition, there are companies and third-party developers who have developed applications to allow smartphones to send out location information, information about how fast you’re are driving, combined with the information of real time traffic to give you the best routes to avoid traffic, or give you information of the nearby gas station, restaurant, bank etc.

While shopping, the loyalty card of the store is combined with your purchase history and social media data to offer you discount coupons and personalized offers based on your loyalty to the store. This is really making shopping more fun, economical and very personalized.

Hospitals, healthcare and fitness

Doctors are maintaining record of patients to keep track of their medical history, for better, quicker and more pin point treatment. While, pediatric units in hospitals are live steaming heartbeats of premature and sick babies in the womb. Combining the information with historically data and based on analysis, doctors are now able to detect infections in babies even before they are born.

Fitness and healthcare have become the biggest market for electronic companies. Most manufacturers, including Nike have are coming out with fitness bands, smartwatches and pedometers etc to collect daily data of a person’s physical routine, calories, sleep patterns and heart rate etc., which is then sent wirelessly to smartphone, doctors and insurance companies to devise  better and more customized healthcare programs.

New course to handle Big Data on Hadoop using R software

New course to handle Big Data on Hadoop using R software

Jigsaw Academy is introducing all new course in big data analytics using R and Hadoop. The course has been specifically designed to provide students’ knowledge and hone their skills to handle big data environment of Hadoop using the R software.

JigsawAcademyIt just been days we learnt of Cloudera and Udactiy partnership to offer open Hadoop and MapReduce courses. Course which have been specially designed to equip students with technical and analytical skills for a brighter career in emerging data market.Following the lead, Jigsaw Academy, a premier online analytics training academy, has introduced new courses in Big Data Analytics using R and Hadoop.

Jigsaw Academy has made a good name in online analytics training. It offers both intermediate and advanced level big data analytics courses. With a vision to extend its roots (as a premier academy), Jigsaw Academy has specifically designed their new course to provide everyone (in need) knowledge and help develop skills needed to deal with big data analytics on Hadoop using R software.

SaritaDigumarti, co-founder at Jigsaw Academy informs,

This new course is specifically designed for those looking to enhance their knowledge and skill sets in Big Data, specifically that of handling the big data environment of Hadoop using R software.

Who is the course for?

Since, Jigsaw Academy thrives on the continuous commitment to expand its offerings, the new course will really help global industry experts (who lack big data handling skills) garner significant expertise in the big data analytics environment. The primary target group for the course, being offered, are analytics experts who are wanting to learn and develop on their big data analytics skills.

It is also beneficial students planning to pursue a career in data science, or for those database professionals who plan to make an entry into the big data analytics industry.

Requirements for enrollment?

To attain an entry into the course, professionals and students are required to have working knowledge of R software. They should have a beginner’sunderstanding of statistics and SQL.

Those not versed with R will have to undergo a spate R skills course, which will be offered by Jigsaw Academy for free.

What to expect in and on completion of the course?

The course can be really beneficial for all the aforementioned type because the instructors at Jigsaw Academy will use real-time big data case studies. This will allows the instructors to showcase and clear the concepts of Hadoop in addition to providing training of application of big data technologies on large volume of data.

What to expect on completion?

  • A working knowledge of Hadoop
  • An ability to analyze big data using R software
  • Complete knowledge of big data analytics
  • And practical application of big data analytics

Via: PRWeb

To bridge market needs Udacity introduces paid big data courses

To bridge market needs Udacity introduces paid big data courses

Data analysts and data scientists are already in big demand. According to various surveys there will be an exponential requirement of big data experts around the globe in next three years, with 190,000 data science experts needed in US alone. Given this demand, data science courses are a good career option. In addition to various free online courses, Udacity has introduced paid courses to impart more refined and superior education. 

Choosing to launch full time experience in data science courses, Udacity recently announced two new initiatives for students. These include Data Science and Big Data Track and Paid Course Enrollment. The idea behind paid courses is to improve the focus on the free online courses that are otherwise very unprofessional.

Udacity paid big data courses

For the new courses, Udacity will make available both a new track and a new personal coach for students who register and enroll for the course by paying a monthly pre-determined fee. The courses comprise of lectures, project and auto-graded exercises in addition to the online content (which is available for free).

Significance of Udacity’s paid course

When asked about the importance of paid course, Udacity informed that enrolling in a paid course is like going to a great class, in contrast to a free online course which is like reading through a book aimlessly.

According to Udacity, the paid enrollment course will begin from January 2014. The charges under the scheme will be charged per month/per course. The good thing about the courses will be that these will continue without any rigid schedule. There will be no fixed start and end dates, so the students will have the flexibility to take as long as they desire to finish a particular course.

The real advantage that students will have with Udacity courses is that during the period of enrollment, students will have access to a personal coach who will assist them through the projects and guide them with advice and feedback. Students will also benefit with verified certificates, which will be presented for the accomplishment at the completion of the final project and exit interview.

Project work

Udacity’s paid enrollment scheme will lay extreme emphasis on project work. This is because a good job takes more than a good resume. Udacity course will thus require lot of project work, which will help students show off their projects as portfolio to their employers. This will increase their opportunity to get employment and enhance their practical knowledge.

Paid enrollment scheme – cost structure

Presently, Udacity has only released information about a few courses. Students who are enrolling in advance are being offered 30% discount. The discount in enrollment is available for four courses which are part of Data Science and Big Data Track. For entry level course the cost is $150 per month (discount available now) and for intermediate courses like Data Wrangling with MongoDB, which will be two month course, the cost is $200 per month (discount available now). In January a couple of more beginner level courses will be added.

Via: I-Programmer