Big Data & Data Engineering MCQ

Explanation

Big Data refers to datasets whose volume, velocity, or variety exceed the capabilities of traditional data processing tools, requiring distributed systems.

Explanation

The classic 3 Vs are Volume (amount of data), Velocity (speed of generation/processing), and Variety (different data types and formats).

Explanation

HDFS (Hadoop Distributed File System) splits large files into blocks and distributes them across a cluster of machines for fault tolerance and parallel access.

Explanation

MapReduce is a programming model that splits a task into a "map" phase (transform/filter) and a "reduce" phase (aggregate), enabling parallel distributed processing.

Explanation

Batch processing handles accumulated data in scheduled chunks, while stream processing handles events in near real-time as they are generated.

Explanation

Data warehouses hold structured, schema-on-write data optimized for reporting, while data lakes store raw structured, semi-structured, and unstructured data cheaply.

Explanation

ETL describes the process of extracting data from sources, transforming it into a usable format, and loading it into a target system like a data warehouse.

Explanation

ELT leverages the processing power of modern warehouses, loading raw data first and transforming it with SQL or compute inside the warehouse afterward.

Explanation

Structured data follows a predefined schema (e.g., tables with rows/columns), making it easy to query with SQL, unlike semi-structured or unstructured data.

Explanation

JSON and XML are semi-structured: they have some organizational structure (tags, keys) but do not enforce a rigid schema like relational tables.

Explanation

NoSQL databases (document, key-value, column-family, graph) typically offer flexible schemas and horizontal scalability, suited for large or varied datasets.

Explanation

Schema-on-read (common in data lakes) stores raw data and applies structure only when queries read it, offering flexibility compared to schema-on-write.

Explanation

A data pipeline automates the flow of data through stages such as ingestion, transformation, and loading, often run on a schedule or triggered by events.

Explanation

OLTP (Online Transaction Processing) systems support fast, frequent writes for operational apps, while OLAP (Online Analytical Processing) systems support complex read-heavy analytics.

Explanation

Parquet stores data column-by-column, enabling efficient compression and allowing queries to read only the columns they need, ideal for analytics.

Explanation

Partitioning organizes data into subsets (e.g., by date or region) so queries can scan only relevant partitions instead of the entire dataset.

Explanation

SQL remains central to data engineering for querying databases, writing transformations in ELT pipelines, and interacting with most warehouses and engines.

Explanation

Data ingestion is the entry point of a pipeline, bringing data from sources like databases, APIs, or files into a system for further processing.

Explanation

Keeping raw data provides a source of truth, enabling reprocessing if transformation logic changes or bugs are found, without re-fetching from the original source.

Explanation

Distributed computing splits work across many connected machines (nodes) so large datasets and computations can be processed in parallel, faster than on one machine.

Explanation

Apache Spark is a distributed computing engine for large-scale batch and stream data processing, known for in-memory computation and speed over disk-based MapReduce.

Explanation

A node is one machine in a cluster, contributing CPU, memory, and storage resources to the overall distributed system.

Explanation

Data quality covers dimensions like accuracy, completeness, consistency, timeliness, and validity — all crucial for trustworthy analytics and decisions.

Explanation

A primary key is a column (or set of columns) whose values uniquely identify each row, ensuring no duplicate identifiers exist.

Explanation

Indexes act like a lookup structure (often a B-tree) that allows the database to find rows quickly without scanning the entire table.

Explanation

Fact tables hold numeric measures (e.g., revenue, quantity) and foreign keys referencing dimension tables, forming the core of star/snowflake schemas.

Explanation

Dimension tables contain descriptive context (who, what, where, when) referenced by foreign keys in fact tables, enabling slicing and filtering of measures.

Explanation

Normalization organizes data to minimize duplication and dependency issues, typically by splitting data into multiple related tables following normal forms.

Explanation

Denormalization trades some redundancy for faster reads by combining related data, which is often desirable in analytical (OLAP) workloads with heavy reads.

Explanation

Data governance establishes who can access, modify, and use data, ensuring compliance, quality, and security across an organization.

Explanation

Metadata provides context about data — such as column names, types, source, and last-updated time — helping users and systems understand and manage datasets.

Explanation

A data catalog helps users discover and understand available datasets across an organization, often including search, lineage, and ownership information.

Explanation

CDC tracks row-level changes in a source database (often via transaction logs) and streams them to downstream systems, enabling near real-time replication.

Explanation

Idempotent pipelines can safely be re-run (e.g., after a failure) without creating duplicate records or corrupting data, which is critical for reliability.

Explanation

Serialization formats like JSON, Avro, and Protobuf convert in-memory data into bytes for storage or network transfer, then deserialize it back later.

Explanation

Compression reduces storage costs and I/O time, which is especially valuable for big data where datasets can be enormous; the trade-off is CPU usage.

Explanation

Schedulers like cron (or more advanced orchestrators) trigger pipeline jobs on a defined schedule, such as running an ETL job every night.

Explanation

Data lineage tracks how data flows and transforms through systems, helping with debugging, impact analysis, and regulatory compliance.

Explanation

Snowflake is a popular cloud-based data warehouse, alongside others like Amazon Redshift and Google BigQuery, used for large-scale analytics.

Explanation

Data engineers design, build, and maintain the systems and pipelines that make reliable data available for analysts, scientists, and applications.

Explanation

DataFrames are built on top of RDDs and add a schema, allowing Spark's Catalyst optimizer to generate more efficient execution plans than raw RDD operations.

Explanation

Operations like map() or filter() are transformations that build a DAG lazily; operations like count() or collect() are actions that trigger execution.

Explanation

By deferring execution, Spark can analyze the full chain of transformations and apply optimizations like predicate pushdown before running any actual computation.

Explanation

Operations like groupBy, join, or repartition require data with the same key to be on the same node, causing data to be moved across the network — a shuffle.

Explanation

When one table is small enough to fit in memory on each executor, broadcasting it avoids shuffling the large table, significantly speeding up the join.

Explanation

Topics are logical channels in Kafka; producers write records to topics and consumers read from them, with topics divided into partitions for scalability.

Explanation

Partitions allow Kafka to parallelize and scale a topic, and each message within a partition has a sequential offset that consumers use to track position.

Explanation

A DAG represents tasks as nodes and dependencies as directed edges with no cycles, allowing the scheduler to determine a valid execution order.

Explanation

Pipelines often need to be retried or backfilled; idempotent tasks (e.g., using overwrite instead of append) ensure re-running produces the same correct result.

Explanation

Window functions (e.g., ROW_NUMBER, RANK, running totals with OVER) compute values across a "window" of rows without collapsing the result like GROUP BY does.

Explanation

SCD Type 2 tracks history by inserting new rows (with start/end dates or a current flag) instead of overwriting, unlike Type 1 which simply overwrites.

Explanation

Combining partitioning (coarse-grained pruning, e.g., by date) with bucketing (hashing rows into fixed buckets by a key) optimizes both scan and join performance.

Explanation

Storing values column-by-column means analytical queries that touch few columns read less data, and similar values compress better than row-oriented storage.

Explanation

Schema evolution allows producers and consumers using different schema versions to remain compatible (e.g., adding optional fields with defaults).

Explanation

Data validation frameworks define expectations (e.g., "column X must not be null") and run automated checks within pipelines to catch issues before they propagate.

Explanation

The medallion architecture progressively refines data: bronze (raw ingest), silver (cleaned and joined), and gold (aggregated, business-ready) layers.

Explanation

CAP theorem describes the trade-off that during a network partition, a system must choose between consistency and availability, while still being partition tolerant.

Explanation

Eventually consistent systems prioritize availability and allow temporary divergence between replicas, converging over time without immediate synchronization.

Explanation

ZooKeeper coordinates distributed processes — managing cluster metadata, leader election, and configuration — though newer Kafka versions reduce this dependency.

Explanation

The driver process runs the main program, builds the execution plan, and sends tasks to executors, which perform the actual computation in parallel.

Explanation

Caching stores intermediate results in memory (or disk) so repeated actions on the same DataFrame avoid recomputation, at the cost of memory usage.

Explanation

Skew occurs when certain keys dominate the data, overloading the tasks/nodes handling them, which can be mitigated with techniques like salting or repartitioning.

Explanation

Exactly-once semantics ensure that despite retries due to failures, each event's effect is applied only once in the output, avoiding duplicates or loss.

Explanation

Watermarks define a threshold for how late data can arrive and still be included in a windowed computation, balancing latency and completeness.

Explanation

Micro-batching divides a continuous stream into small time-based batches, balancing the low latency of true streaming with the efficiency of batch processing.

Explanation

Flink is built around native, continuous event-at-a-time streaming, while Spark traditionally used micro-batches (though Spark has improved with continuous processing options).

Explanation

Materialized views cache the results of expensive queries on disk, trading storage and refresh overhead for much faster read performance on repeated queries.

Explanation

SCD Type 2 commonly adds versioning columns (start/end dates, current flag, or version number) so historical attribute values can be queried as of a point in time.

Explanation

Data mesh shifts from a centralized data platform team to domain-oriented teams owning and serving their data as discoverable, well-governed "data products".

Explanation

A schema registry ensures producers and consumers serialize/deserialize messages using compatible schema versions, preventing data corruption from mismatches.

Explanation

Backpressure mechanisms signal upstream stages to slow down or buffer when downstream stages are overwhelmed, preventing resource exhaustion or data loss.

Explanation

In columnar formats, selecting only needed columns lets the engine skip reading irrelevant column data, reducing I/O, cost, and query time significantly.

Explanation

Checkpoints persist offsets and state to durable storage so that on failure, a streaming job can resume from the last checkpoint instead of starting over.

Explanation

Cloud object stores provide cheap, durable, scalable storage decoupled from compute, letting multiple engines (Spark, Presto, etc.) query the same data.

Explanation

Vertical scaling ("scale up") increases capacity of one node, with hard limits; horizontal scaling ("scale out") distributes load across many nodes, common in big data systems.

Explanation

Event time is based on when data was generated, allowing correct results despite network delays or out-of-order arrival, unlike processing time which depends on system clock.

Explanation

Deduplication removes redundant copies of the same event, which commonly arise from retry logic in at-least-once delivery systems, ensuring accurate aggregates.

Explanation

YARN separates resource management from processing logic, allowing multiple processing frameworks (MapReduce, Spark) to share cluster resources efficiently.

Explanation

coalesce() is more efficient for reducing partitions since it minimizes data movement, while repartition() performs a full shuffle and can both increase and decrease partition count.

Explanation

When a query filters on the partitioning column (e.g., WHERE date = '2024-01-01'), the engine can skip irrelevant partitions entirely, greatly improving performance.

Explanation

Catalyst applies rule-based and cost-based optimizations to transform a logical plan into an efficient physical plan, such as reordering filters or choosing join strategies.

Explanation

Tungsten focuses on low-level CPU/memory efficiency, including binary in-memory data formats and generating JVM bytecode for whole stages to avoid interpretation overhead.

Explanation

AQE uses runtime statistics (actual partition sizes after a shuffle) to adapt the plan, e.g., switching a sort-merge join to a broadcast join or reducing skewed partitions.

Explanation

Kafka transactions let a producer atomically write to multiple partitions and commit consumer offsets together, so a failure and retry does not produce duplicates downstream.

Explanation

Rebalancing redistributes partitions across active consumers in a group to maintain parallel processing, but can cause brief processing pauses ("stop-the-world" in older protocols).

Explanation

Lambda runs parallel batch and stream layers (risking logic divergence), while Kappa simplifies by treating all data as a stream, reprocessing from the log when needed.

Explanation

Lakehouse formats add a transactional metadata layer over files in object storage, enabling ACID guarantees, concurrent writes, schema evolution, and historical versioning ("time travel").

Explanation

A transaction log records each commit as a new snapshot referencing specific data files, so querying "as of" a prior version reconstructs the table state from that snapshot.

Explanation

Z-ordering interleaves values from multiple columns so that files contain a narrower range of values for each, allowing more files to be skipped when filtering on those columns.

Explanation

The "small file problem" from streaming writes increases metadata overhead and slows scans; periodic compaction jobs merge small files into optimally sized larger files.

Explanation

Sort-merge join scales well for large datasets via sorting, while shuffle hash join can be faster when one side fits in memory after shuffling, avoiding the sort step.

Explanation

Columnar formats like Parquet store per-column statistics (min/max) per row group, so the reader can skip entire row groups that cannot satisfy a filter, drastically reducing I/O.

Explanation

Dictionary encoding stores unique values once and references them by small integer codes, yielding large compression gains for low-cardinality columns like country or status.

Explanation

True end-to-end exactly-once requires coordination across all three stages; the sink often needs idempotent writes (e.g., upserts keyed by a unique ID) or two-phase commit support.

Explanation

Log-based CDC (used by Debezium) reads the database's internal replication/transaction log, capturing changes with minimal performance impact compared to polling-based approaches.

Explanation

Salting splits a single overloaded key into several "salted" sub-keys distributed across partitions, then the results are combined, balancing the workload across tasks.

Explanation

A checkpoint must atomically capture both the processing state and the corresponding input offsets so that recovery resumes from a consistent point, avoiding gaps or duplicates.

Explanation

To avoid rewriting all historical files immediately, formats handle schema evolution by reading missing columns as null or default values for older files, deferring physical rewrites.

Explanation

CBO relies on table/column statistics (cardinality, distinct counts, histograms) to estimate costs of alternative plans, e.g., choosing join order or join strategy.

Explanation

Tuning shuffle partitions balances memory per task against per-task overhead; too few causes large tasks that may OOM, while too many causes excessive scheduling overhead.