1

Unit-1

Data Engineering

Data engineering is the process of designing and building systems that let
people collect and analyze raw data from multiple sources and formats.
These systems empower people to find practical applications of the data,
which businesses can use to thrive.

Data engineering is a field within data science that focuses on designing, constructing, and
maintaining the systems and infrastructure that allow for the processing, storage, and
retrieval of large volumes of data. Data engineers are responsible for developing pipelines
and architectures to extract, transform, and load (ETL) data from various sources into data
warehouses, data lakes, or other storage systems. They work closely with data scientists and
analysts to ensure that data is available, accessible, and usable for analysis and decision-
making purposes.

Key responsibilities of data engineers include:

1. Data Pipeline Development: Designing and implementing data pipelines to move
data from various sources to its destination, ensuring efficiency, reliability, and
scalability.
2. Data Storage and Management: Setting up and maintaining databases, data
warehouses, and data lakes to store and organize large volumes of structured and
unstructured data.
3. Data Transformation: Performing data transformation tasks such as cleaning,
aggregating, and structuring data to make it suitable for analysis and reporting.
4. Data Quality and Governance: Implementing processes and standards to ensure data
quality, integrity, and security throughout its lifecycle.
5. Performance Optimization: Optimizing data processing and storage systems for
performance, scalability, and cost-effectiveness.
6. Collaboration: Working closely with cross-functional teams including data scientists,
analysts, and business stakeholders to understand data requirements and deliver
solutions that meet their needs.

Data engineering is essential for organizations that deal with large volumes of data, such as
e-commerce companies, social media platforms, and financial institutions, as it enables
them to leverage data effectively for business insights, decision-making, and innovation.
2
3

Data Engineering Tools and Skills

Data engineers use many different tools to work with data. They use a
specialized skill set to create end-to-end data pipelines that move data from
source systems to target destinations.

Data engineers work with a variety of tools and technologies, including:

 ETL Tools: ETL (extract, transform, load) tools move data between
systems. They access data, then apply rules to “transform” the data
through steps that make it more suitable for analysis.
 SQL: Structured Query Language (SQL) is the standard language for
querying relational databases.
 Python: Python is a general programming language. Data engineers
may choose to use Python for ETL tasks.
 Cloud Data Storage: Including Amazon S3, Azure Data Lake Storage
(ADLS), Google Cloud Storage, etc.
 Query Engines: Engines run queries against data to return answers.
Data engineers may work with engines like Dremio Sonar, Spark,
Flink, and others.

Data Engineers skills and activites

Data engineers require a combination of technical skills, domain

knowledge, and practical experience to effectively design and
implement data solutions. Here's a list of key skills and activities
typically associated with data engineering:
4

Technical Skills:

1. Programming Languages: Proficiency in languages commonly

used in data engineering such as Python, SQL, Scala, Java, or R.

2. Data Querying and Manipulation: Strong understanding of SQL

for querying and manipulating data in relational databases.

3. Data Processing Frameworks: Experience with frameworks like

Apache Spark, Apache Flink, Apache Beam, or Apache Kafka for
distributed data processing.

4. Data Storage Technologies: Knowledge of various data storage

technologies including relational databases (e.g., PostgreSQL,
MySQL), NoSQL databases (e.g., MongoDB, Cassandra), data
warehouses (e.g., Amazon Redshift, Google BigQuery), and
distributed file systems (e.g., Hadoop HDFS, AWS S3).

5. ETL Tools: Familiarity with Extract, Transform, Load (ETL) tools

such as Apache Airflow, Apache NiFi, Talend, or Informatica for
building data pipelines.

6. Data Modeling: Understanding of data modeling concepts and

experience in designing and implementing data models for
different use cases.

7. Cloud Platforms: Proficiency in cloud platforms such as AWS,

Azure, or Google Cloud Platform, including services like AWS
Glue, Azure Data Factory, Google Cloud Dataflow, and related
storage and compute services.

8. Version Control: Knowledge of version control systems like Git

for managing code and collaboration with other team
members.
5

Domain Knowledge:
1. Data Architecture: Understanding of data architecture
principles and best practices for designing scalable, efficient,
and reliable data solutions.

2. Domain Expertise: Familiarity with the specific domain or

industry the organization operates in, including relevant data
sources, data types, and analytics requirements.

Practical Experience and Activities:

1. Data Pipeline Development: Designing, implementing, and

maintaining data pipelines to extract, transform, and load data
from various sources to storage and analytics platforms.

2. Data Quality Management: Implementing processes and

standards to ensure data quality, integrity, and consistency
throughout the data lifecycle.

3. Performance Optimization: Optimizing data processing and

storage systems for performance, scalability, and cost-
effectiveness.

4. Monitoring and Maintenance: Monitoring data pipelines and

infrastructure for performance issues, errors, and anomalies,
and performing troubleshooting and maintenance tasks as
needed.

5. Collaboration: Working closely with cross-functional teams

including data scientists, analysts, software engineers, and
business stakeholders to understand requirements and deliver
data solutions that meet business needs.
6

6. Documentation and Communication: Documenting data

pipelines, systems, and processes, and effectively
communicating technical concepts and solutions to non-
technical stakeholders.

7. Continuous Learning: Keeping up-to-date with emerging

technologies, tools, and best practices in data engineering
through self-study, training, and professional development
opportunities.

Data engineers inside organization

Within an organization, data engineers typically serve several key

functions and play integral roles in enabling data-driven decision-
making, facilitating analytics, and supporting various business
operations.

Here's a breakdown of how data engineers contribute within an

organization:

1. Data Infrastructure Development: Data engineers are responsible for designing,

building, and maintaining the infrastructure required to store, process, and analyze
data.

This includes setting up databases, data warehouses, data lakes, and data pipelines
to ensure that data is accessible, reliable, and scalable.

2. Data Pipeline Implementation: Data engineers develop and maintain data pipelines
to extract, transform, and load (ETL) data from various sources into storage and
analytics platforms.

These pipelines automate the process of ingesting data, ensuring efficiency,

reliability, and timeliness in data delivery.

3. Data Quality Assurance: Data engineers implement processes and standards to

ensure data quality, integrity, and consistency throughout the data lifecycle.

This involves performing data validation, cleansing, and enrichment to improve the
accuracy and reliability of data for analysis and reporting purposes.
7

4. Performance Optimization: Data engineers optimize data processing and storage

systems for performance, scalability, and cost-effectiveness.

This includes tuning database queries, optimizing data pipeline workflows, and
leveraging cloud services to maximize resource utilization and minimize latency.

5. Collaboration with Data Scientists and Analysts: Data engineers work closely with
data scientists, analysts, and other stakeholders to understand data requirements,
develop solutions, and deliver insights that drive business value.

This collaboration involves translating business needs into technical requirements

and implementing data solutions that support analytical workflows.

6. Security and Compliance: Data engineers implement security measures and

compliance controls to protect sensitive data and ensure regulatory compliance.

This includes managing access controls, encryption, and auditing mechanisms to

safeguard data privacy and integrity.

7. Documentation and Knowledge Sharing: Data engineers document data

infrastructure, pipelines, and processes to ensure clarity and transparency for other
team members and stakeholders.

They also share knowledge and best practices with the broader organization to foster
a culture of data-driven decision-making and continuous improvement.

8. Training and Skill Development: Data engineers invest in their professional

development by staying abreast of emerging technologies, tools, and best practices
in data engineering.

They may also provide training and mentorship to junior team members to enhance
the overall skill set within the organization.
8

Data engineering lifecycle

The data engineering lifecycle encompasses the processes and activities

involved in designing, building, deploying, and maintaining data solutions
within an organization.

While specific methodologies and frameworks may vary depending on the

organization and project requirements, the data engineering lifecycle typically
includes the following stages:

1. Requirement Analysis:
 Define business objectives and data requirements.
 Identify key stakeholders and their needs.
 Gather and document functional and non-functional requirements
for the data solution.

2. Data Ingestion:
 Identify data sources and understand data formats and structures.
 Develop mechanisms to extract data from source systems, which
may include databases, APIs, files, or streaming platforms.
 Implement data ingestion pipelines to bring data into the data
ecosystem.

3. Data Processing:
 Transform raw data into a format suitable for analysis and storage.
 Cleanse, validate, and enrich data to ensure quality and
consistency.
 Perform aggregations, calculations, and other data
transformations as needed.

4. Data Storage:
 Select appropriate storage technologies based on data volume,
velocity, and variety.
 Design and implement data storage solutions such as databases,
data warehouses, data lakes, or cloud storage services.
 Define data schemas and structures to optimize storage and
retrieval efficiency.
9

5. Data Integration:
 Integrate data from multiple sources to create a unified view of
the data landscape.
 Merge and reconcile data to ensure consistency across different
data sets.
 Implement data integration pipelines to automate the process of
combining and harmonizing data.

6. Data Analysis and Exploration:

 Provide tools and interfaces for data exploration and analysis, such
as dashboards, visualization tools, or query interfaces.
 Enable self-service access to data for data scientists, analysts, and
business users.
 Support ad-hoc queries, exploratory analysis, and hypothesis
testing.

7. Data Governance and Security:

 Implement data governance policies and procedures to ensure
data quality, privacy, and compliance with regulations.
 Define access controls, encryption mechanisms, and auditing
capabilities to protect sensitive data.
 Monitor and enforce data governance policies throughout the
data lifecycle.

8. Performance Optimization:
 Optimize data processing and storage systems for performance,
scalability, and cost-effectiveness.
 Tune database queries, optimize data pipelines, and leverage
caching and indexing techniques to improve performance.
 Monitor system metrics and performance indicators to identify
bottlenecks and areas for optimization.

9. Deployment and Operations:

 Deploy data solutions to production environments and ensure
reliability, availability, and scalability.
 Monitor system health and performance, and proactively identify
and address issues.
 Implement backup and disaster recovery mechanisms to protect
against data loss and system failures.
10

10.Maintenance and Evolution:

 Maintain and evolve data solutions over time to accommodate
changing business requirements and technological advancements.
 Perform routine maintenance tasks such as software updates,
security patches, and performance tuning.
 Continuously improve data engineering processes and practices
based on feedback and lessons learned.

Throughout the data engineering lifecycle, collaboration and communication

between data engineers, data scientists, analysts, business stakeholders, and IT
teams are essential to ensure the success of data initiatives and the delivery of
valuable insights to the organization.
11

The Data lifecycle vs Data Engineering Lifecycle

Data Lifecycle:
The Data Lifecycle encompasses the entire journey of data from its creation or acquisition to
its disposal or archival. It typically includes the following stages:

1. Data Generation/Capture: Data is created or collected from various sources such as

sensors, applications, transactions, or interactions.
2. Data Ingestion: Data is ingested into the organization's systems or data infrastructure
from its source locations.
3. Data Storage: Data is stored in appropriate storage systems, such as databases, data
warehouses, data lakes, or cloud storage.
4. Data Processing: Data is processed, transformed, and analyzed to derive insights or
to prepare it for further analysis.
5. Data Usage: Processed data is used for various purposes such as reporting, analytics,
machine learning, decision-making, or operational tasks.
6. Data Archiving: Data that is no longer actively used may be archived for long-term
storage or compliance purposes.
7. Data Deletion/Disposal: Data that has reached the end of its lifecycle and is no
longer needed is securely deleted or disposed of according to data retention policies
and regulations.

Data Engineering Lifecycle:

The Data Engineering Lifecycle, on the other hand, focuses specifically on the processes and
activities involved in designing, building, deploying, and maintaining data solutions within an
organization. It encompasses the following stages:

1. Requirement Analysis: Gathering and understanding business requirements and data

needs.
2. Data Ingestion: Extracting, transforming, and loading (ETL) data from various sources
into storage and analytics platforms.
3. Data Processing: Cleansing, transforming, and enriching data to ensure quality and
consistency.
4. Data Storage: Designing and implementing data storage solutions to store and
manage data effectively.
5. Data Integration: Integrating data from multiple sources to create a unified view of
the data landscape.
6. Data Analysis and Exploration: Providing tools and interfaces for data exploration,
analysis, and visualization.
7. Data Governance and Security: Implementing policies and procedures to ensure
data quality, privacy, and compliance.
8. Performance Optimization: Optimizing data processing and storage systems for
performance, scalability, and cost-effectiveness.
9. Deployment and Operations: Deploying data solutions to production environments
and ensuring reliability and availability.
10. Maintenance and Evolution: Maintaining and evolving data solutions over time to
accommodate changing requirements and advancements.
12

While the Data Engineering Lifecycle is a subset of the broader Data Lifecycle, it focuses
specifically on the technical aspects of managing and processing data within an organization,
from its ingestion to its utilization and maintenance.

Both lifecycles are essential for effectively managing data assets and deriving value from
data-driven initiatives.

Designing good Data Architecture

Designing a good data architecture involves creating a framework that supports the
organization's data needs efficiently, securely, and flexibly. Here are some key principles and
steps to consider when designing a robust data architecture:

Understand Business Requirements:

Gather Requirements: Work closely with stakeholders to understand business

objectives, data sources, use cases, and performance requirements.
Identify Data Needs: Determine the types of data required (structured, semi-structured,
unstructured) and their volume, velocity, and variety.

Define Data Layers:

Raw Data Layer: Store raw, unprocessed data as it is ingested from source systems. This
layer serves as the foundation for downstream processing and analysis.
Refined Data Layer: Transform and cleanse raw data into a structured format suitable for
analysis. Apply business rules, data quality checks, and enrichment to improve data
quality and usability.
Aggregated Data Layer: Aggregate and summarize refined data for reporting,
dashboarding, and analytics. Precompute aggregations to improve query performance
for commonly used metrics.

Choose Appropriate Data Storage Technologies:

Relational Databases: Use for structured data requiring ACID (Atomicity, Consistency,
Isolation, Durability) transactions and complex querying.

NoSQL Databases: Opt for document-oriented, key-value, column-family, or graph

databases for semi-structured or unstructured data with flexible schemas.

Data Warehouses: Implement for storing and analyzing structured data at scale, with
optimized query performance and support for analytics workloads.
Data Lakes: Utilize for storing large volumes of raw and unstructured
data, providing flexibility for data exploration and analysis.
13

Establish Data Integration and ETL Processes:

Data Ingestion: Develop robust processes for ingesting data from various sources,
including databases, APIs, files, and streaming platforms.

ETL Pipelines: Design Extract, Transform, Load (ETL) pipelines to process and transform
data from its raw form to refined and aggregated formats.
Real-time Data Processing: Implement streaming data pipelines for real-time processing
and analysis of high-velocity data streams.

Ensure Data Governance and Security:

Data Quality Management: Implement data quality checks, validations, and cleansing
routines to maintain data integrity and accuracy.
Access Control: Define roles, permissions, and access controls to restrict access to
sensitive data and ensure compliance with privacy regulations.

Data Encryption: Encrypt data at rest and in transit to protect against unauthorized
access and data breaches.
Audit Logging: Maintain audit logs to track data access, modifications, and usage for
compliance and security purposes.

Enable Scalability and Performance:

Horizontal Scalability: Design systems that can scale horizontally by adding more
resources or nodes to handle increasing data volumes and user loads.

Optimized Query Performance: Indexing, partitioning, and denormalization can enhance

query performance, especially for large datasets.
Caching and Materialized Views: Utilize caching and materialized views to precompute
and store frequently accessed data for faster retrieval.

Plan for Disaster Recovery and High Availability:

Backup and Restore: Implement regular backups of data to prevent data loss in the
event of hardware failures, human errors, or disasters.

Failover and Redundancy: Design systems with redundancy and failover mechanisms to
ensure high availability and minimize downtime.

Continuously Monitor and Iterate:

Performance Monitoring: Monitor system performance, resource utilization, and query

execution times to identify bottlenecks and optimize system performance.
Feedback Loop: Collect feedback from users and stakeholders to identify areas for
improvement and iterate on the data architecture accordingly.
14

Unit-2
Data generation in source system

Data generation in source systems refers to the process of creating or collecting data
from various sources such as applications, sensors, devices, databases, logs, or external
systems. This data is typically generated as a result of interactions, transactions, events,
or processes occurring within an organization or its external environment. Here's a
breakdown of key aspects related to data generation in source systems:

Types of Source Systems:

Transactional Systems: These systems capture data generated from day-to-day business
operations such as sales transactions, customer interactions, financial transactions, and
inventory movements.
Operational Systems: Operational systems include applications and software used to
manage business processes, such as Enterprise Resource Planning (ERP), Customer
Relationship Management (CRM), and Human Resource Management (HRM) systems.

Sensors and IoT Devices: Data can be generated by sensors, IoT devices, and connected
machinery, providing real-time information about physical processes, environmental
conditions, and equipment performance.
Log Files and Events: Applications, servers, and network devices generate log files and
event streams containing information about system activities, errors, and performance
metrics.
External Data Sources: Organizations may also collect data from external sources such as
third-party APIs, data feeds, social media platforms, and public datasets for enrichment
or integration with internal data.

Data Generation Process:

Event Trigger: Data generation is often triggered by events such as user interactions,
system updates, sensor readings, or scheduled processes.
Data Capture: Upon occurrence of an event, relevant data attributes are captured and
recorded by the source system. This data may include structured data stored in
databases, semi-structured data in log files, or unstructured data in text documents or
multimedia files.

Data Transformation: In some cases, data may undergo initial transformation or

preprocessing within the source system to standardize formats, cleanse data, or enrich it
with additional context before being stored or transmitted.
15

Characteristics of Generated Data:

Volume: The amount of data generated can vary significantly depending on factors such
as the size of the organization, the frequency of events, and the granularity of data
captured.
Velocity: Data may be generated at varying speeds, ranging from real-time streaming
data to batch updates or periodic snapshots.

Variety: Data generated by source systems can be structured, semi-structured, or

unstructured, and may include text, numbers, dates, images, videos, or geospatial
information.
Veracity: Ensuring the accuracy, reliability, and consistency of generated data is crucial
for maintaining data quality and integrity throughout its lifecycle.

Challenges and Considerations:

Data Consistency: Ensuring consistency and coherence of data across multiple source
systems and data sources can be challenging, especially in distributed or heterogeneous
environments.
Data Quality: Maintaining data quality standards requires implementing validation
checks, error handling mechanisms, and data cleansing procedures to address
inaccuracies, duplications, or inconsistencies in generated data.
Data Security: Protecting sensitive data from unauthorized access, disclosure, or
tampering is essential to maintain data privacy and compliance with regulatory
requirements.

Scalability: As data volumes and sources grow, source systems must be scalable to
handle increased data processing and storage requirements efficiently.
Integration Complexity: Integrating data from diverse source systems and formats may
require implementing middleware, ETL processes, or data integration platforms to
facilitate data movement and transformation.

Effective management of data generation in source systems is foundational to building

robust data pipelines, analytics workflows, and decision support systems that enable
organizations to harness the value of their data assets for strategic insights and
operational excellence.
16

How is Data created

Data is created through various processes and activities across different domains and
industries. Here are some common ways in which data is generated:

1. Human Interaction: Data is often created through human interactions with digital
systems, such as entering information into forms, applications, or databases. For
example, when a user registers on a website, fills out a survey, or makes a purchase
online, data is generated capturing their actions and preferences.

2. Transactions: Many data records are created as a result of business transactions. This
includes sales transactions, financial transactions, inventory movements, and other
operational activities within organizations. For instance, each purchase made at a
retail store generates transactional data capturing details such as the item
purchased, quantity, price, and payment method.

3. Sensor Readings: Data is generated by sensors and IoT devices that monitor physical
processes, environmental conditions, and equipment performance. These sensors
capture various types of data such as temperature, pressure, humidity, motion,
location, and vibration, providing real-time insights into the operation of machinery,
infrastructure, and systems.

4. Log Files and Events: Applications, servers, and network devices generate log files
and event streams containing information about system activities, errors, warnings,
and performance metrics. These logs record events such as user logins, database
queries, application errors, resource utilization, and network traffic, which are
valuable for troubleshooting, monitoring, and auditing purposes.

5. Social Media and Web Interactions: Data is created through user interactions on
social media platforms, websites, and digital channels. This includes posts,
comments, likes, shares, clicks, page views, and other engagement metrics that
provide insights into user behavior, preferences, and trends.

6. Machine-generated Data: Automated systems and processes generate data as part

of their operation. This includes data generated by automated workflows, scripts,
algorithms, and software applications. For example, data generated by
recommendation engines, predictive models, automated trading systems, and
autonomous vehicles.

7. Biological and Scientific Instruments: Data is generated by scientific instruments,

medical devices, and research equipment used in laboratories, healthcare settings,
and scientific experiments. This includes genetic sequencing data, medical imaging
data, climate data, and other scientific measurements and observations.
17

Explain Source system and Source systems practical details:

What is a Source System?

A source system refers to any system or application that generates or stores data, serving as
the point of origin for data collection within an organization. These systems can be diverse,
ranging from transactional databases to log files, IoT devices, external APIs, and more.

Practical Details about Source Systems:

1. Types of Source Systems:

 Transactional Systems: These systems handle day-to-day business operations
and generate transactional data. Examples include ERP systems, CRM
systems, point-of-sale (POS) systems, and online transaction processing
(OLTP) databases.
 Operational Systems: These systems support various operational processes
within an organization. They can include inventory management systems,
supply chain systems, manufacturing execution systems (MES), and human
resource management systems (HRMS).

 Log Files and Event Streams: Applications, servers, and network devices
generate log files and event streams containing information about system
activities, errors, and performance metrics. Examples include web server logs,
application logs, and system logs.
 IoT Devices: Internet of Things (IoT) devices such as sensors, actuators, and
smart devices generate data about physical processes, environmental
conditions, and equipment status. This data can include sensor readings,
telemetry data, and device status updates.
 External Data Sources: Organizations often integrate data from external
sources such as third-party APIs, data feeds, social media platforms, and
public datasets. These external data sources can provide additional context,
enrichment, or validation for internal data.

2. Data Formats and Structures:

 Source systems may generate data in various formats, including structured,

semi-structured, and unstructured data.
 Structured data typically follows a predefined schema and is organized into
tables or records with well-defined fields and data types. Examples include
data stored in relational databases or spreadsheets.

 Semi-structured data lacks a strict schema but has some organizational

structure, such as JSON, XML, or CSV files.
 Unstructured data has no predefined structure and can include text
documents, images, videos, audio recordings, and other multimedia files.
18

3. Data Integration Challenges:

 Data Variety: Source systems may generate data in different formats and
structures, making it challenging to integrate and reconcile data from
disparate sources.
 Data Volume: High volumes of data generated by source systems can strain
data processing and storage resources, requiring scalable solutions for
ingestion, processing, and analysis.
 Data Velocity: Real-time or near-real-time data streams from source systems
may require streaming data processing and analytics capabilities to handle
high data velocity.

 Data Quality: Ensuring data quality and consistency across source systems is
essential for maintaining the integrity and reliability of data pipelines and
analytics workflows.

4. Data Ingestion and Integration:

 Organizations employ various techniques for ingesting and integrating data

from source systems into their data infrastructure.
 This can include batch processing techniques for periodic data loads, real-
time streaming data ingestion for continuous data streams, and change data
capture (CDC) mechanisms for capturing incremental changes to data.

5. Data Governance and Security:

 Source systems often contain sensitive or proprietary data that must be

protected from unauthorized access, disclosure, or tampering.
 Implementing data governance policies, access controls, encryption
mechanisms, and audit logging helps ensure data security and compliance
with regulatory requirements.

6. Data Lifecycle Management:

 Source systems contribute to the data lifecycle by generating data at the
creation stage. Proper data lifecycle management involves managing data
from its creation through various stages, including ingestion, storage,
processing, analysis, and archival or disposal.

In practice, effective management of source systems involves understanding the

characteristics of each source system, implementing appropriate data integration strategies,
ensuring data quality and security, and aligning data governance practices with
organizational objectives and regulatory requirements.

Additionally, leveraging modern data integration platforms and tools can streamline the
process of ingesting, integrating, and managing data from diverse source systems.
19

Undercurrents and their impact on source system

"Undercurrents" in the context of source systems typically refer to hidden or underlying

factors that influence the behavior, performance, and characteristics of these systems.

These undercurrents can have a significant impact on the operation, management, and
effectiveness of source systems. Here are some common undercurrents and their potential
impact:

1. Legacy Systems: Many organizations rely on legacy systems that have been in place
for years or even decades. These systems may have outdated technology,
architecture, or infrastructure, making them difficult to maintain, integrate, or
extend.

The undercurrent of legacy systems can hinder agility, scalability, and innovation
within an organization, affecting the ability to adapt to changing business needs and
technological advancements.

2. Data Silos: Data silos occur when data is fragmented and isolated within different
systems or departments, making it challenging to access, share, and analyze across
the organization. The undercurrent of data silos can lead to duplication of effort,
inconsistent data, and missed opportunities for collaboration and insights. It can also
hinder the implementation of holistic data governance and management practices.

3. Technical Debt: Technical debt refers to the accumulation of suboptimal design,

implementation, or maintenance decisions made over time. These shortcuts or
compromises may result in increased complexity, technical limitations, and higher
maintenance costs.

The undercurrent of technical debt can impede system performance, reliability, and
scalability, as well as hinder the ability to innovate and adapt to changing
requirements.

4. Data Quality Issues: Poor data quality, including inaccuracies, incompleteness, and
inconsistency, can undermine the reliability and trustworthiness of data generated by
source systems.

The undercurrent of data quality issues can lead to erroneous insights, incorrect
decisions, and decreased confidence in data-driven initiatives. It may also result in
additional overhead for data cleansing, validation, and reconciliation efforts.
20

5. Organizational Culture and Practices: Organizational culture and practices play a

significant role in shaping how source systems are managed, used, and maintained.
The undercurrent of organizational culture can influence factors such as
communication, collaboration, accountability, and risk tolerance.

It can impact the adoption of new technologies, methodologies, and best practices,
as well as the effectiveness of data governance and management initiatives.

6. Regulatory Compliance: Source systems may be subject to various regulatory

requirements, industry standards, and data privacy regulations that govern how data
is collected, processed, and stored.

The undercurrent of regulatory compliance can introduce complexities, constraints,

and legal obligations that impact system design, operation, and maintenance. Failure
to comply with regulatory requirements can result in legal and financial
consequences for organizations.

7. External Dependencies: Source systems may rely on external dependencies such as

third-party APIs, data providers, cloud services, or infrastructure components. The
undercurrent of external dependencies introduces risks related to service availability,
reliability, and security. It can also impact data integration, interoperability, and
vendor management efforts within an organization.

Addressing these undercurrents requires a holistic approach that involves identifying root
causes, implementing remediation strategies, and fostering a culture of continuous
improvement and adaptation.

This may include modernizing legacy systems, breaking down data silos, managing technical
debt, improving data quality processes, fostering collaboration and communication, ensuring
compliance with regulations, and mitigating risks associated with external dependencies.

By addressing these undercurrents effectively, organizations can enhance the resilience,

agility, and effectiveness of their source systems in support of their broader business
objectives.
21

Unit-3

Raw ingredients of data storage

The raw ingredients of data storage encompass the fundamental components and
technologies required to store, organize, and manage data effectively. These ingredients
form the foundational infrastructure upon which data storage solutions are built. Here are
some key raw ingredients of data storage:

1. Storage Media:
 Hard Disk Drives (HDDs): Traditional mechanical storage devices that use
spinning platters to store data magnetically.
 Solid State Drives (SSDs): Storage devices that use flash memory to store
data, offering faster access times and better reliability compared to HDDs.
 Tape Drives: Magnetic tape-based storage devices used for long-term archival
storage and backup purposes.

2. Storage Systems:
 Storage Area Networks (SANs): High-speed networks dedicated to providing
access to block-level storage devices, typically used in enterprise
environments.
 Network Attached Storage (NAS): File-level storage devices connected to a
network, often used for file sharing and centralized data storage.
 Direct Attached Storage (DAS): Storage devices directly attached to a single
server or computer, providing local storage capacity.

3. Storage Protocols:
 SCSI (Small Computer System Interface): A standard protocol used for
communication between storage devices and computers.
 Fibre Channel: A high-speed networking protocol commonly used in SAN
environments for connecting storage devices to servers.
 NFS (Network File System) and SMB (Server Message Block): Protocols used
for sharing files over a network, commonly associated with NAS devices.

4. Data Replication and Redundancy:

 RAID (Redundant Array of Independent Disks): A data storage technology
that combines multiple physical disks into a single logical unit to improve
performance, reliability, or both.
 Replication: The process of duplicating data across multiple storage devices
or locations to ensure redundancy and fault tolerance.
22

5. Data Management Software:

 File Systems: Software responsible for organizing and managing files on

storage devices, providing features such as file allocation, directory structure,
and access control.
 Volume Managers: Software used to manage storage volumes, partitions, and
logical drives, allowing for flexible allocation and utilization of storage
resources.
 Backup and Recovery Software: Tools and applications used to create
backups of data, restore data in case of data loss or corruption, and manage
backup policies and schedules.
 Data Deduplication and Compression Software: Software solutions that
reduce storage space by eliminating duplicate data or compressing data to
reduce storage requirements.

6. Storage Management Tools:

 Storage Resource Management (SRM) Tools: Tools used to monitor, analyze,

and optimize storage resources, including capacity planning, performance
monitoring, and reporting.
 Storage Virtualization: Technologies that abstract physical storage resources
into virtualized pools, allowing for centralized management and allocation of
storage capacity.
 Data Tiering and Hierarchical Storage Management (HSM): Techniques used
to automatically move data between different storage tiers based on access
frequency, cost, and performance requirements.

7. Security and Access Control Mechanisms:

 Encryption: Techniques used to encrypt data at rest and in transit to protect

against unauthorized access and data breaches.
 Access Control Lists (ACLs) and Permissions: Mechanisms used to control
access to data and storage resources, specifying which users or groups have
permission to read, write, or modify data.
23

Data Storage Systems

Data storage systems encompass a variety of technologies and architectures designed to

store, organize, and manage data efficiently. These systems range from traditional on-
premises storage solutions to cloud-based storage services. Here are some common types of
data storage systems:

1. Direct Attached Storage (DAS):

 DAS refers to storage devices directly attached to a single server or computer,
typically using interfaces like SATA, SAS, or USB.
 Examples include internal hard drives, external hard drives, and solid-state
drives (SSDs).
 DAS is suitable for small-scale deployments or environments where data
needs are relatively simple and localized.

2. Network Attached Storage (NAS):

 NAS devices are file-level storage devices connected to a network, providing
centralized storage accessible to multiple clients and applications.
 They use file-level protocols such as NFS (Network File System) or SMB
(Server Message Block).
 NAS solutions are commonly used for file sharing, data backup, media
storage, and serving as storage targets for virtualization environments.

3. Storage Area Network (SAN):

 SANs are high-speed networks dedicated to providing access to block-level
storage devices.
 They typically use Fibre Channel or iSCSI protocols for block-level access.
 SANs offer centralized storage that can be shared among multiple servers,
enabling features like data replication, snapshots, and storage virtualization.
 SANs are commonly used in enterprise environments for mission-critical
applications, database storage, and virtualization deployments.

4. Object Storage:
 Object storage systems organize data as objects, each with a unique identifier
and metadata.
 They are designed for scalability, durability, and efficient access to large
volumes of unstructured data.
 Object storage is well-suited for use cases such as cloud storage, content
delivery, media streaming, and data archival.
 Examples of object storage solutions include Amazon S3, Google Cloud
Storage, and OpenStack Swift.
24

5. Cloud Storage Services:

 Cloud storage services provide scalable and on-demand storage resources
accessible over the internet.
 They offer features such as pay-as-you-go pricing, scalability, data
redundancy, and global accessibility.
 Cloud storage solutions include object storage, file storage, block storage, and
archival storage options.
 Major cloud providers offering storage services include Amazon Web Services
(AWS), Microsoft Azure, Google Cloud Platform (GCP), and IBM Cloud.

6. Software-Defined Storage (SDS):

 SDS abstracts storage resources from physical hardware and provides storage
services through software.
 It enables flexibility, automation, and scalability in managing storage
infrastructure.
 SDS solutions can run on commodity hardware or virtualized environments
and often support integration with cloud storage services.
 Examples of SDS solutions include Ceph, GlusterFS, and VMware vSAN.

7. Hyper-Converged Infrastructure (HCI):

 HCI combines storage, compute, and networking resources into a single,
integrated system.
 It eliminates the need for separate storage arrays and simplifies deployment,
management, and scaling of infrastructure.
 HCI solutions often include features like software-defined storage,
virtualization, and automation.
 Leading HCI vendors include VMware vSAN, Nutanix, and Microsoft Azure
Stack HCI.

8. Backup and Data Protection Systems:

 Backup and data protection systems provide mechanisms for creating,
managing, and storing backups of critical data.
 They include features such as backup scheduling, data deduplication,
encryption, and replication.
 Backup solutions may leverage traditional tape-based systems, disk-based
backup appliances, or cloud-based backup services.
25

Data Engineering storage abstraction

storage abstraction in data engineering involves creating a layer of software that hides the
complexities of different storage systems behind a unified interface. Here's how it works:
1. Unified Interface:
 Storage abstraction provides a consistent set of APIs, libraries, or query languages
that data engineers can use to interact with data, regardless of the underlying
storage technology.
 For example, instead of writing different code to access data from a relational
database, a NoSQL database, and a cloud storage service, data engineers can use the
same set of commands or functions.

2. Data Formats and Serialization:

 Storage abstraction often involves standardizing data formats and serialization
methods to ensure interoperability across different storage systems.
 Common data formats like JSON, Avro, or Protocol Buffers are used to serialize data,
allowing it to be stored, retrieved, and processed uniformly.

3. Connectivity and Integration:

 Storage abstraction layers include connectors, drivers, or adapters that enable
seamless integration with various storage systems.
 These connectors translate data requests from the unified interface into storage-
specific commands understood by the underlying storage systems, and vice versa.

4. Metadata Management:
 Metadata catalogs store information about the structure, location, and lineage of
data stored across different storage systems.
 Data engineers can query these catalogs to discover, explore, and understand the
data assets available in the environment, regardless of their physical storage location
or format.

5. Data Movement and ETL:

 Storage abstraction facilitates data movement and ETL processes by providing
abstraction layers for data ingestion, transformation, and loading operations.
 ETL pipelines can be designed to work with abstracted data sources and sinks,
enabling flexibility in orchestrating data workflows and pipelines across
heterogeneous storage environments.

6. Data Lifecycle Management:

 Storage abstraction includes mechanisms for managing the lifecycle of data stored
across different storage systems, such as data retention policies, archiving, and data
expiration.
 Data lifecycle management ensures that data is stored, processed, and retained
according to business requirements and compliance regulations, regardless of the
underlying storage technology.
7. Integration with Data Processing Frameworks:
 Storage abstraction layers are often integrated with data processing frameworks and
libraries such as Apache Spark, Apache Flink, or Apache Beam.
 These frameworks provide connectors, adapters, or APIs for interacting with
abstracted storage systems, allowing for efficient data processing and analytics
workflows.
26

In technical terms, storage abstraction simplifies the management and utilization of data by
providing a uniform interface for interacting with diverse storage technologies.

It enables data engineers to focus on developing data-driven applications and analytics solutions
without being tied to the specifics of underlying storage implementations, thereby promoting agility,
scalability, and interoperability in data management processes

Big trends in storage

1. Cloud Storage Dominance:

 Cloud storage continues to grow in popularity due to its scalability, flexibility,
and cost-effectiveness.

Organizations are increasingly adopting cloud storage solutions from

providers such as Amazon Web Services (AWS), Microsoft Azure, and Google
Cloud Platform (GCP) to store and manage their data.

 Hybrid and multi-cloud strategies are also becoming more prevalent, allowing
organizations to leverage multiple cloud providers for redundancy, flexibility,
and vendor lock-in avoidance.

2. Object Storage Adoption:

 Object storage is gaining traction as a preferred method for storing
unstructured data such as images, videos, documents, and log files. Object
storage systems offer scalability, durability, and simplicity compared to
traditional file and block storage solutions.

 Leading object storage platforms include Amazon S3, Azure Blob Storage,
Google Cloud Storage, and open-source solutions like Ceph and MinIO.

3. Software-Defined Storage (SDS):

 Software-defined storage (SDS) continues to evolve, providing organizations
with greater flexibility and control over their storage infrastructure.
 SDS abstracts storage resources from physical hardware and provides storage
services through software, enabling features such as autom
ation, orchestration, and policy-based management.

 SDS solutions like Ceph, GlusterFS, and VMware vSAN are gaining popularity
for their scalability, agility, and cost-effectiveness.

4. NVMe and SSD Adoption:

 Non-Volatile Memory Express (NVMe) and Solid-State Drives (SSDs) are
becoming increasingly prevalent in storage systems, offering higher
performance, lower latency, and greater reliability compared to traditional
hard disk drives (HDDs).
27

 NVMe SSDs are particularly well-suited for high-performance workloads, such

as database acceleration, real-time analytics, and virtualized environments.

5. Hyper-Converged Infrastructure (HCI):

 Hyper-Converged Infrastructure (HCI) solutions are gaining momentum as
organizations seek to simplify their IT infrastructure and streamline
operations.
 HCI combines compute, storage, and networking resources into a single,
integrated system, reducing complexity, improving scalability, and enhancing
resource utilization.

 Leading HCI vendors include VMware vSAN, Nutanix, and Microsoft Azure
Stack HCI.

6. Data Management and Analytics Integration:

 There is a growing emphasis on integrating data storage with data
management and analytics platforms to enable real-time insights and
decision-making.

 Storage solutions are increasingly incorporating features such as data tiering,

data deduplication, compression, and encryption to optimize storage
efficiency and security.

 Integration with data processing frameworks like Apache Spark, Apache Flink,
and Apache Hadoop enables organizations to analyze data directly in storage
without needing to move it to separate analytics platforms.

7. Edge Storage and Computing:

 With the proliferation of IoT devices and edge computing applications, there
is a growing demand for storage solutions at the edge.

 Edge storage solutions enable data to be processed and stored closer to the
point of generation, reducing latency and bandwidth requirements for edge
applications.
 Leading cloud providers are offering edge storage services and edge
computing platforms to support edge use cases such as autonomous vehicles,
smart cities, and industrial IoT.