Unit - 1

What is data?
 Dictionary Definition:
The quantities, characters, or symbols on which
operations are performed by a computer, which may
be stored and transmitted in the form of electrical
signals and recorded on magnetic (audio tape), optical
(CD), or mechanical recording media (Phonographic
disc)
What is big data?
 It is a collection of data that is huge in volume and yet
growing exponentially with time.

 It is a data with so large size and complexity that none

of traditional data management tools can store it or
process it efficiently.

 Big data is also a data but with huge size.

Definition
 Big data is high-volume, and or / high velocity
information assets that demand cost-
effective, innovative forms of information processing
for enhanced insight and decision making.
 - Gartner IT Glossary
 The huge data is of the order of tera (10^12)bytes, Peta
(10^15) bytes or Zeta (10^21) bytes.
Explanation of Big Data definition
Big Data Definition
Why is Big Data important?
 Using the data from any source and analyzing it, we
can find answers that
 Streamline resource management
 Improve operational efficiencies
 Optimize product development
 Drive new revenue and growth opportunities
 Enable smart decision making
Big data enables to accomplish
business related tasks
 Determine the root causes of failures, issues and
defects in near- real time (industrial usage)
 Spotting anomalies faster and more accurately than
human eye (healthcare usage)
 Recalculating entire risk portfolios in minutes
(investment / finance sector)
 Detect fraudulent behavior before it affects your
organization.
Some examples of big data
 The NYSE generates about one terabyte of new trade
data per day.
 55 billion messages and 4.5 billion photos are
exchanged on wattsapp every day
 300 hours of video is uploaded every minute
 Every minute user sends 31.25 million messages and
watch 2.77 million videos
 There are around 40,000 search queries googled each
second.
Types of Big Data
 Structured
 Semi-Structured
 Unstructured
Types of Big Data
Big Data

Unstructured (80%)

Structured (10%)

Semi Structured (10%)

Un-structured Data
10 Characteristics of Big Data
Values Visualization
Volume Data
Getting value
Data Size clustering, sunbursts
out of data
Velocity
Vulnerability
Speed at which
Security Concerns
data is generated
The 10 V’s
Variety Volatility
Different types of of Big data
Data governance
data
Variability
Dynamic Evolving Validity
behavior in data Veracity Data quality check
science Confidence or Trust in
data
Data accuracy
Unstructured data for Analytics
 Business Documents
 Emails
 Social Media
 Customer feedback
 Webpages
 Open-ended survey responses
 Images, Audio and Video

 Importance of unstructured data analysis for businesses:

 Improve the customer experience
 Discover gaps in the market and innovate
 Listen to your customers
How to Analyze unstructured Data
 Choose the End Goal
 Define a clear set of measurable goals.

 Collect Relevant Data

 Focus on the source of data

 Clean Data
 Reduce noise
 Eliminate unwanted information

 Implement Technology
 NoSQL databases
 Data visualization using Tableau, Google data studio
Unstructured Data Analytics
Data • Association Rule
Mining mining
• Regression Analysis
• Collaborative filtering
Dealing Text
with Mining
unstructure
d data

NLP

Noisy
Text
Analysis
Unstructured Data Analytics Tools
 MonkeyLearn – Used for Text Analytics.
 This tool makes it simple to clean, label and visualize customer
feedback
 Word Clouds - textual data visualization which allows anyone to
see in a single glance the words which have the highest frequency
within a given body of text
 Listen to customer’s voice – open surveys and emails.
 Aspect is based on sentiment analysis
 Amazon AWS
 Microsoft Azure
 IBM Cloud
The Advantages of Deploying Big
Data
 Better Decision making

 Cost Reduction

 Newer Products and redevelopment of the old

 Risk Analysis

 Collection of Data
Industries using Big Data
 CA technology have done a global study in which clearly the benefits of
Big data outweigh the obstacles in implementation
 The percentage of organizations that plan to and already have
implemented a big data project is 84%
 Acquision has increased to 54%, revenue has improved by 88%.

 hiQ: It specializes in ‘people analytics’.

 SumAI: Helps businesses optimize their social media campaigns with
the help of one single chart.
 Splunk: Visual analytics
 Alteryx: Combines structured and unstructured data from a number
of sources and stores it in one database. Spacial, predictive and
statistical analysis tasks are done on this data.
How big data is used in different
industries
 Media and entertainment:
 Companies like Hulu and Netflix work with big data to
analyze user tendencies, preferred content, trends in
consumption.
 Lot of services like spotify are coming up with new
revenue models to increase profits
 Ads are targeted more strategically thanks to big data
analytics software.
Finance
 Shift from Manual trading to trading backed by
technology

 These models analyze big data to make

 accurate enter / exit trade decisions,
 minimize risk using machine learning and
 guage market sentiment using opinion mining
Healthcare
 With predictive analytics , big data can predict
negative health events that senior citizens would
experience from home care.
 This reduced visits by 73% and 64% amongst
chronically ill patients
 Big data can identify disease trends based on
demographics, geographies, socio economics and
other factors
Education
 Improve learning management. Tracking how much
time learners spend on tasks, tests, and exams helps to
customize curricula efficiently.
 Improve students’ performance. Leveraging data about
learners’ performance helps educators develop
personalized learning paths.
 Provide data-driven decision-making.
 Predict learning outcomes.
 Use big data to reduce dropout rates
Retail
 Enhance Service Quality

 Optimize Price

 Manage Supply Chain

 Identify Potential Risks

 Forecast demand
Manufacturing
 Quality Assurance

 Supply Chain Optimization

 Improving Throughput and Yield

 Less downtime

 Greater Customer Service

Big Data Challenges
Big Data Challenges
 1. Lack of Knowledge Professionals
 To run these large data tools, companies need skilled
data professionals. (data scientists, data analysts and
data engineers)
 Solution : Big data tools are used by professionals who are not
data science experts but have the basic knowledge.
 This saves a lot of money for the companies.
 2. Lack of proper understanding of Massive Data
 Employees not knowing how to store sensitive data.
 Solution: Data workshops and seminars must be held at
companies for everybody.
Big Data Challenges
 3. Data Growth Issues:
 One of the biggest challenges is to store the huge data.
 Solution: Compression is used to reduce the size of data
stored.
 De-duplication removes the duplicate and unwanted data
 Data Tiering stores data in different data tiers.(public
clouds, private cloud and flash storage)
 4. Confusion while Big data tool selection
 Companies are confused on which tool to select for Data
analysis and storage? Hbase, Cassandra etc.
 Solution: Hire experts who know which tools to use.
Big Data Challenges
 5. Integrating data from a spread of sources
 Data in corporation comes from various sources like
social media pages, ERP applications, customer logs etc.
 Solution: Data integration problems are solved by purchasing
proper tools.
 6. Securing Data
 Companies can lost a lot of revenue due to a stolen
record or a knowledge breach.
 Solution: cyber-security professionals guard their data. Other
steps include encryption, identity and access control,
implementation of end point security real-time security
monitoring
Assignment Questions
1. What is Big Data ?
2. Explain the types of data. Also briefly mention the sources
of each types of data along with examples.
3. Why is big data important. How does it help businesses
and briefly describe its usage across various
domains(industry, retail, healthcare, manufacturing, educ
ation …)
4. Briefly describe the characteristics of big data.
5. Describe the types of analytics and mention the sources of
unstructured data used in analytics.
6. Mention some tools used in analytics
7. Discuss the Big Data challenges briefly