Google Practice Questions, Discussions & Exam Topics by our Authors
You need to set access to BigQuery for different departments within your company. Your solution should comply with the following requirements:
* Each department should have access only to their data.
* Each department will have one or more leads who need to be able to create and update tables and provide them to their team...
Analyzing the options in detail:
- Option A: Create a dataset for each department. Assign the department leads the role of OWNER, and assign the data analysts the role of WRITER on their dataset.
- Why not selected: This option does not meet the requirement that data analysts should only be able to query data and not modify it. The WRITER role allows users to create and update tables, which is not suitable for data analysts who only need to query the data. The OWNER role for department leads is appropriate for managing datasets, but assigning WRITER to data analysts is not a good fit for the role they need.
- Option B: Create a dataset for each department. Assign the department leads the role of WRITER, and assign the data analysts the role of READER on their dataset.
- Why selected: This option meets the requirements perfectly. By creating a separate dataset for each department, you can effectively separate access by department. The WRITER role for department leads allows them to create and update tables, which is necessary for them to manage their department's data. The READER role for data analysts ensures they can query the data without the ability to modify or create tables, fulfilling the requirement that data analysts should not be able to modify data. This is a balanced, secure solution that m...
Author: RadiantJaguar56 · Last updated Jul 15, 2026
You operate a database that stores stock trades and an application that retrieves average stock price for a given company over an adjustable window of time. The data is stored in Cloud Bigtable where the datetime of the stock trade is the beginning of the row key. Your application has thousands of concurrent users, and you...
Analyzing the options in detail:
- Option A: Change the row key syntax in your Cloud Bigtable table to begin with the stock symbol.
- Why not selected: Changing the row key to start with the stock symbol will lead to hot-spotting in Cloud Bigtable. Hot-spotting occurs when too many requests are directed to a small range of row keys, which are grouped based on similar characteristics. Since stock symbols are likely limited in number compared to time-based row keys, this can create imbalanced access patterns, reducing performance. It also doesn’t directly address the issue of efficiently querying stock prices over time ranges.
- Option B: Change the row key syntax in your Cloud Bigtable table to begin with a random number per second.
- Why selected: This option helps address the performance problem by randomizing the row keys, effectively distributing writes and read requests across Cloud Bigtable’s nodes, which prevents hot-spotting. This solution leverages Cloud Bigtable's ability to scale horizontally, making it efficient even with high-concurrency scenarios. By introducing randomness in the row keys, data is distributed more evenly, improving performance for high read/write throughput, especially with thousands of concurrent users.
- Option C: Change the data pipeline to use BigQuery for storing stock trades, and update your application.
...
Author: Vikram · Last updated Jul 15, 2026
You are operating a Cloud Dataflow streaming pipeline. The pipeline aggregates events from a Cloud Pub/Sub subscription source, within a window, and sinks the resulting aggregation to a Cloud Storage bucket. The source has consistent throughput. You want to monitor an alert on behavio...
Analyzing the options in detail:
- Option A: An alert based on a decrease of subscription/num_undelivered_messages for the source and a rate of change increase of instance/storage/used_bytes for the destination.
- Why not selected: A decrease in num_undelivered_messages typically means that messages are being successfully delivered and processed by the pipeline. However, an increase in instance/storage/used_bytes for the destination could indicate a bottleneck, leading to an overflow of data or storage issues, but this alert doesn't directly address a situation where the pipeline is not processing data. This combination of metrics doesn't efficiently capture problems with data processing or identify issues that prevent the pipeline from consuming messages.
- Option B: An alert based on an increase of subscription/num_undelivered_messages for the source and a rate of change decrease of instance/storage/used_bytes for the destination.
- Why selected: This is the most effective choice. An increase in num_undelivered_messages for the source indicates that the pipeline is not consuming data as expected. If the number of undelivered messages grows, it means the pipeline is falling behind, which could be due to a bottleneck in processing or a failure in the pipeline. Additionally, a decrease in instance/storage/used_bytes for the destination indicates that the pipeline might not be writing data to Cloud Storage as expected. These combined alerts would effectively monitor and al...
Author: Grace · Last updated Jul 15, 2026
You currently have a single on-premises Kafka cluster in a data center in the us-east region that is responsible for ingesting messages from IoT devices globally.
Because large parts of globe have poor internet connectivity, messages sometimes batch at the edge, come in all at once, and cause a spike in load on your
Kafka cluster...
In this scenario, the goal is to build a scalable, cloud-native architecture to handle global IoT device messages efficiently and to avoid the challenges of handling spikes in load that come with poor internet connectivity in certain regions.
Evaluation of the options:
A) Edge TPUs as sensor devices for storing and transmitting the messages
- Not a suitable choice: Edge TPUs are primarily designed for machine learning tasks at the edge, such as inference workloads. While they are optimized for processing ML models, they are not suitable for handling or managing message ingestion, especially in IoT systems. This option would not address the issue of message batching or global scalability in this context.
B) Cloud Dataflow connected to the Kafka cluster to scale the processing of incoming messages
- Not a suitable choice: While Cloud Dataflow is a fully managed service that can scale, it is primarily focused on stream processing. If the existing Kafka cluster is experiencing load spikes due to large batch messages coming in all at once, scaling the Kafka cluster alone may not fully resolve the issue of addressing connectivity challenges. Moreover, relying on Cloud Dataflow directly connected to the Kafka cluster doesn’t address the core issue of efficient ingestion and distribution of messages to the right services before they hit the Kafka cluster.
C) An IoT gateway connected to Cloud Pub/Sub, with Cloud Dataflow to read and process the messages from Cloud Pub/Sub
- The best option:
- Why it works: This solution addresses the global connectivity issues by utilizing Cloud Pub/Sub, which is designed to decouple message ingestion from processing and can buffer messages as they come in. Pub/Sub can efficiently handle varying message rates and act as a "gate" for messages before they get to the processing layer (Cloud Da...
Author: StarlightBear · Last updated Jul 15, 2026
You decided to use Cloud Datastore to ingest vehicle telemetry data in real time. You want to build a storage system that will account for the long-term data growth, while keeping the costs low. You also want to create snapshots of the data periodically, so that you can make a point-in-time (PIT) recovery, or clone a copy of the data for Clo...
Let's analyze the options based on the requirements: long-term data growth, low-cost storage, periodic snapshots for point-in-time (PIT) recovery, and archiving for long-term storage.
Evaluation of the options:
A) Use managed export, and store the data in a Cloud Storage bucket using Nearline or Coldline class.
- This is a good option.
- Why it works: Cloud Datastore provides a managed export feature that can be used to export your data in a consistent and reliable manner. Once exported, the data can be stored in Cloud Storage in the Nearline or Coldline storage classes. These storage classes are specifically designed for long-term archival and low-cost storage, making them an ideal choice for keeping snapshots of the data at a low cost.
- Key advantages:
- Cost-effective: Nearline and Coldline storage classes are cost-effective for storing infrequently accessed data, which is typical for long-term archival of snapshots.
- Scalable: Cloud Storage can scale to accommodate large amounts of data growth over time.
- Easy to restore: If you need to restore or perform PIT recovery, the snapshots are easily accessible from Cloud Storage.
B) Use managed export, and then import to Cloud Datastore in a separate project under a unique namespace reserved for that export.
- Not ideal:
- Why it's not recommended: This method will create a new Cloud Datastore instance with exported data in a separate project, which involves additional complexity. You will still have the costs associated with managing and storing the exported data in Datastore, and this approach does not directly address long-term archiving or low-cost storage.
- Key disadvantages:
- Costs: Storing the exported data back in Datastore can incur ongoing costs, especially as data grows.
- No archival advantage: This method doesn’t leverage the cost-effective archival options like Cloud Storage Nearline/Coldline.
C) Use managed export, and then import the data into a BigQuery table created just for that export, and delete temporary export files.
- Not ideal:
- Why it's not recommended: While BigQuery is great for analyzing large datasets, it may not be the most cost-effective solution for storing long-term snapshots, particularly if you don’t need to query the data frequently. Importing the exported data into BigQuery adds complexity and may incur ongoing storage costs, particularly for long-term storage of raw data.
- Key disadvantages:
- Cost: BigQuery charges for storage and queries, so storing snapshots in BigQuery could be more expensive than Cloud Storage (especially Coldline/Nearline storage).
- Not optimized...
Author: Maya2022 · Last updated Jul 15, 2026
You need to create a data pipeline that copies time-series transaction data so that it can be queried from within BigQuery by your data science team for analysis.
Every hour, thousands of transactions are updated with a new status. The size of the initial dataset is 1.5 PB, and it will grow by 3 TB per day. The data is heavily structured, and your data science team will build machi...
Let's evaluate the strategies based on the requirements:
Key Requirements:
- Data volume: Initial size of 1.5 PB, growing by 3 TB per day.
- Time-series data: The data is updated every hour with a new status.
- Performance: Need a solution that maximizes performance and usability for the data science team.
- Machine learning models: Data will be used for machine learning, requiring efficient querying and data processing.
Evaluation of the Options:
A) Denormalize the data as much as possible.
- This is a good option.
- Why it works: Denormalization reduces the complexity of joins when querying the data. Since time-series data typically involves aggregating and filtering over time, having a denormalized schema can significantly improve query performance in BigQuery. Denormalizing also helps to reduce the number of JOIN operations, which can be expensive in terms of performance, especially when dealing with large datasets.
- Key advantages:
- Improved query performance: Denormalized data makes it easier to query because all relevant data is contained in a single table or fewer tables, reducing the need for complex joins.
- Machine learning suitability: Denormalized data is often easier to work with in machine learning models since it avoids unnecessary processing steps that could slow down feature engineering or training processes.
B) Preserve the structure of the data as much as possible.
- Not ideal in this scenario:
- Why it's not recommended: While preserving the structure might seem logical for some types of datasets, time-series data can become complex if left overly normalized, especially when frequently updated. Time-series data often requires efficient querying across time windows, and preserving too much structure (i.e., excessive normalization) can increase the number of joins and thus decrease query performance in BigQuery.
- Key disadvantages:
- Query inefficiency: If the data is highly normalized, queries will require multiple joins, which can hurt performance in BigQuery, especially with a growing dataset.
- Complexity: More normalized structures may complicate both data loading and querying, which can negatively affect the usability and speed needed for machine learning models.
C) Use BigQuery UPDATE to further reduce the size of the dataset.
- Not ideal in this scenario:
- Why it's not recommended: While `UPDATE` operations in BigQuery are possible, they can be slow and expensive when working with large datasets like 1.5 PB, especially with time-series data that updates frequently. BigQuery is optimized for append-only workloads rather than updates, and using `UPDATE` frequently can result in higher costs and slower performance.
- Key disadvantages:
- Performance and cost: Updates are less efficient for large datasets ...
Author: Oscar · Last updated Jul 15, 2026
You are designing a cloud-native historical data processing system to meet the following conditions:
* The data being analyzed is in CSV, Avro, and PDF formats and will be accessed by multiple analysis tools including Dataproc, BigQuery, and Compute
Engine.
* A batch pipeline moves daily data.
* Performance is ...
Evaluation of the Options:
A) Create a Dataproc cluster with high availability. Store the data in HDFS, and perform analysis as needed.
- Not ideal for this scenario:
- Why it's not recommended: While Dataproc is a powerful tool for distributed processing, using HDFS within Dataproc may not be the best choice here for several reasons. HDFS requires management of clusters and underlying infrastructure, which increases complexity. Additionally, HDFS is not as easy to integrate with multiple tools like BigQuery and Compute Engine directly, which could limit flexibility in accessing data.
- Key disadvantages:
- Management overhead: Managing HDFS requires monitoring, scaling, and ensuring high availability, which contradicts the goal of maximizing availability and minimizing operational complexity.
- Complex integration: Although Dataproc can process data in HDFS, integrating it seamlessly with other tools like BigQuery and Compute Engine might not be as straightforward as using Cloud Storage.
B) Store the data in BigQuery. Access the data using the BigQuery Connector on Dataproc and Compute Engine.
- Not ideal for this scenario:
- Why it's not recommended: BigQuery is optimized for analytical workloads, but it’s not designed for storing raw files in formats like CSV, Avro, or PDF, especially if these files need to be frequently accessed and analyzed directly in their raw format. While BigQuery can handle structured data efficiently, it is not a good choice for storing diverse file types that don’t fit into its tabular model. Additionally, using BigQuery as a storage solution could lead to unnecessary costs and complexity in querying raw files.
- Key disadvantages:
- Not suitable for raw file formats: BigQuery is best for structured data (e.g., tables), but CSV, Avro, and PDF are raw file formats and aren't inherently compatible with BigQuery’s structure.
- Higher cost and complexity: Storing raw files in BigQuery is not cost-effective, and querying raw formats would require additional steps like loading or transforming the data into a structured form.
C) Store the data in a regional Cloud Storage bucket. Access the bucket directly using Dataproc, BigQuery, and Compute Engine.
- Good option:
- Why it works: Cloud Storage is a highly available and scalable solution that supports various file formats such as CSV, Avro, and PDF. A regional Cloud Storage bucket offers high availability within a specific region, which is suffi...
Author: MysticJaguar44 · Last updated Jul 15, 2026
You have a petabyte of analytics data and need to design a storage and processing platform for it. You must be able to perform data warehouse-style analytics on the data in Google Cloud and expose the ...
Key Requirements:
- Petabyte-scale dataset: The dataset is large and requires efficient storage and processing.
- Data warehouse-style analytics: The need to perform analytics indicates a structured data approach with efficient querying and reporting.
- Exposure for batch analysis: The data must be accessible as files for batch processing by other cloud providers.
Let's evaluate the options:
A) Store and process the entire dataset in BigQuery.
- Not ideal for this scenario:
- Why it's not recommended: While BigQuery is an excellent solution for performing data warehouse-style analytics, storing an entire petabyte of data directly in BigQuery may become expensive and inefficient, especially if the dataset needs to be frequently exported for use by tools in other cloud providers. BigQuery is optimized for structured, analytical queries and can handle large datasets, but it might not be the most cost-effective choice for storing the entire dataset if the data also needs to be exposed as files for external tools.
- Key disadvantages:
- Cost: Storing large amounts of data in BigQuery can become expensive, especially as data grows and queries increase.
- Not optimized for file-based access: While BigQuery can export data, it is not designed to serve large datasets as files for batch processing tools, which is a critical requirement in this case.
B) Store and process the entire dataset in Bigtable.
- Not ideal for this scenario:
- Why it's not recommended: Bigtable is optimized for high-throughput, low-latency, and scalable NoSQL data storage. It is suitable for operational data, time-series data, or use cases that require fast reads and writes across large datasets. However, it is not a data warehouse solution and does not support complex analytics like SQL-based querying that is typically required for data warehouse workloads. Also, exposing data in file formats for batch processing by other cloud providers is not a native use case for Bigtable.
- Key disadvantages:
- Not suitable for analytics: Bigtable is not designed for data warehouse-style analytics. It lacks the SQL querying capabilities that are necessary for complex data analysis.
- Not optimized for file-based access: Bigtable is a database service, not a file storage system, so it does not natively support exposing data as files for batch processing by external tools.
C) Store the full dataset in BigQuery, and store a compressed copy of the data in a Cloud Storage bucket.
- This is a good option:
- Why it works: BigQuery is ideal for data warehouse-style analytics, as it supports complex SQL queries and can handle petabyte-scale ...
Author: CrystalWolfX · Last updated Jul 15, 2026
You work for a manufacturing company that sources up to 750 different components, each from a different supplier. You've collected a labeled dataset that has on average 1000 examples for each unique component. Your team wants to implement an app to help warehouse workers recognize incoming components based on a photo of the...
To decide which approach to take, let’s break down the pros and cons of each option, considering the key factors such as time constraints, the scale of the dataset, ease of implementation, and the specific requirements for a Proof-of-Concept (PoC) that can be deployed quickly.
A) Use Cloud Vision AutoML with the existing dataset.
Pros:
- Quick setup: Cloud Vision AutoML allows you to upload the full dataset and create a custom model easily, reducing the need for extensive manual setup.
- Scalability: Can handle a large dataset, as it's designed to scale to different sizes (including 750 components).
- Accuracy: The model can learn from the dataset effectively and may provide high-quality predictions for the various components.
Cons:
- Cost: Using Cloud Vision AutoML can be expensive, especially for large datasets like yours with 750 components. The cost increases as you scale up.
- Data Dependency: Although the model will likely perform well, there’s no guarantee that it will perform optimally right away since the tool isn’t as customizable as training a model from scratch.
When to use: This is ideal if you want a relatively quick and straightforward solution without worrying too much about customization or cost for the PoC.
B) Use Cloud Vision AutoML, but reduce your dataset twice.
Pros:
- Faster training time: Reducing the dataset size will lower the training time, potentially making the model training faster and easier to implement for a PoC.
- Reduced cost: A smaller dataset will reduce the training and usage costs, which can help in controlling expenses for the PoC.
Cons:
- Reduced performance: By halving your dataset, you risk losing the richness and variety of the data, which may lead to a model that doesn't generalize as well or performs poorly on certain components.
- Less representation: With a smaller dataset, the model may not learn as well from all 750 components and may struggle with some edge cases.
When to use: This option might be suitable if you need to deploy a PoC quickly and can tolerate some potential loss in accuracy or performance. However, this will not be ideal for the long term, especially as the number of components is large.
C) Use Cloud Vision API by providing custom labels as recognition hints.
Pros:
- Fast and easy to implement: You can use the Cloud Vis...
Author: Olivia Johnson · Last updated Jul 15, 2026
You are working on a niche product in the image recognition domain. Your team has developed a model that is dominated by custom C++ TensorFlow ops your team has implemented. These ops are used inside your main training loop and are performing bulky matrix multiplications. It currently takes up to several days to train...
To select the best option, let’s assess each choice based on the need for significant training time reduction, the nature of the custom C++ TensorFlow ops, and keeping costs manageable.
A) Use Cloud TPUs without any additional adjustment to your code.
Pros:
- High performance: TPUs are designed for high-throughput matrix operations, especially for deep learning workloads. They excel at operations like large matrix multiplications, which should, in theory, speed up training significantly.
- Low cost (per operation): TPUs are generally more cost-efficient for training deep learning models compared to GPUs or CPUs, particularly for large-scale workloads.
Cons:
- Compatibility with custom ops: Since your model uses custom C++ TensorFlow ops, there's a high chance they aren't directly compatible with TPUs. TPUs rely on TensorFlow's predefined operations and may not efficiently support custom ops unless specifically adapted.
- No immediate performance boost for custom ops: Even though TPUs are great for general tensor computations, custom C++ ops could introduce bottlenecks if they cannot be efficiently executed on the TPU, making it less effective for your use case without modification.
When to use: TPUs are ideal if you're using standard TensorFlow operations, but your custom ops likely make this approach less effective unless you refactor your ops for TPU compatibility.
B) Use Cloud TPUs after implementing GPU kernel support for your custom ops.
Pros:
- Leverage TPU benefits with custom ops: By implementing GPU kernel support for your custom ops, you ensure that they are optimized for TPU execution. This makes your custom operations potentially much faster, as TPUs can handle large matrix operations with incredible speed.
- Speed-up with low cost: Once optimized, TPUs can accelerate training significantly, and costs remain low when compared to GPUs and CPUs.
Cons:
- Complex implementation: This option requires modifying your custom ops to be TPU-compatible, which is time-consuming and may require substantial changes to your code.
- Technical overhead: The complexity of ensuring GPU kernel support for custom ops can add development overhead, especially if you lack experience with TensorFlow’s TPU-specific APIs.
When to use: This option is suitable if you are willing to invest time into adapting your custom ops for TPU compatibility and need a long-term performance boost. It's best for significant performance gains in large-scale training but may be too complex for quick results.
C) Use Cloud GPUs after implementing GPU kernel support for your cu...
Author: Lucas · Last updated Jul 15, 2026
You work on a regression problem in a natural language processing domain, and you have 100M labeled examples in your dataset. You have randomly shuffled your data and split your dataset into train and test samples (in a 90/10 ratio). After you trained the neural network and evaluated your model on a test set, you discover that the root-m...
To determine the best approach for improving your model's performance, let’s assess the current situation and evaluate the options based on the problem and symptoms you’ve shared:
Current Situation:
- Training vs. Test RMSE discrepancy: Your model shows significantly higher RMSE on the training set compared to the test set. This suggests that the model may not be overfitting, since overfitting typically results in better performance on the training set (lower error) and worse performance on the test set (higher error). Instead, this could be indicative of underfitting or a mismatch in model performance between the training and test sets.
A) Increase the share of the test sample in the train-test split.
Pros:
- Potential reduction in overfitting: By allocating more data to the training set, you might improve the model’s learning ability. However, in your case, the training error is already high, indicating a problem with the model's ability to capture patterns, which is typically a case of underfitting, not overfitting.
Cons:
- Doesn't address the root issue: This would not solve the underlying issue. The key issue appears to be the model's difficulty in fitting the training data well, so increasing the test set size won't directly address this problem.
- Loss of test data: Increasing the training set size could leave less data for testing, reducing the ability to effectively evaluate model performance.
When to use: This option could be considered if you suspect the training set is too small to learn meaningful patterns. However, in this case, increasing the test size doesn’t seem relevant.
B) Try to collect more data and increase the size of your dataset.
Pros:
- Improved model generalization: Having more data could help improve model performance, particularly if the current dataset is not representative of the full range of input-output relationships. In general, more data can help the model learn better.
Cons:
- Not the root cause of the issue: The problem you described (higher RMSE on the training set) points more to a potential model underfitting or an issue with the model architecture rather than the dataset size. More data won't help much if the model is fundamentally incapable of capturing the patterns in the existing data.
- Time and effort required: Collecting more data can be a resource-intensive task, and it's not guaranteed to solve the issue if the model’s structure isn’t suited to the problem.
When to use: This option would be useful if you suspect the dataset lacks sufficient variety or scale. However, this doesn't seem to be the main cause of your issue based on the symptoms described.
C) Try out regularization techniques (e...
Author: Sofia2021 · Last updated Jul 15, 2026
You use BigQuery as your centralized analytics platform. New data is loaded every day, and an ETL pipeline modifies the original data and prepares it for the final users. This ETL pipeline is regularly modified and can generate errors, but sometimes the errors are detected only after 2 weeks. You need to provide a method to recover ...
To solve this problem, let's analyze each option in terms of recovery from errors, storage optimization, and BigQuery’s capabilities. The key requirements are:
1. Error detection and recovery: You need a system to recover from errors detected after a couple of weeks.
2. Storage optimization: Backups should be efficient in terms of storage costs, as the data is loaded daily and grows over time.
A) Organize your data in a single table, export, and compress and store the BigQuery data in Cloud Storage.
Pros:
- Simple backup process: Exporting data to Cloud Storage is a straightforward process.
- Compression reduces storage: Compressing the exported data reduces the overall storage footprint.
Cons:
- Not optimal for recovery: If the data corruption or error is detected after a period, restoring from a single backup might be challenging. You would have to restore the entire dataset, and this could result in overwriting valid data that was loaded after the backup.
- No fine-grained recovery: If the error occurs during the ETL process, recovery might not be granular enough to revert only the corrupted data without affecting the rest.
When to use: This option could work if you're fine with periodically backing up the entire dataset, but it doesn't offer a flexible way to restore only the problematic data or parts of it.
B) Organize your data in separate tables for each month, and export, compress, and store the data in Cloud Storage.
Pros:
- Monthly backups: Organizing data into monthly tables provides better granularity for backups. If an issue occurs, you can restore the data from the month before the issue.
- Storage efficiency: Exporting and compressing monthly data into Cloud Storage can still help reduce storage costs, especially if you’re only backing up the data for the month.
Cons:
- Not ideal for long-term recovery: While this solution offers better granularity than a single table, restoring from Cloud Storage is not as efficient as restoring directly within BigQuery. It could also be slow and complicated to manually manage these backups over time.
- Manual process: You would still need to manage the process of exporting and compressing the data, which could introduce additional overhead.
When to use: This is a reasonable solution if you can afford to manage monthly backups manually and have the flexibility to restore data via Cloud Storage. However, it doesn't offer the most seamless way to recover corrupted data from within BigQuery itself.
C) Organize your data in separate tables for each month, and duplicate your data on a separate dataset in BigQuery.
Pros:
- Data duplication within BigQuery: Storing backups within BigQuery itself makes it easier to restore the data as needed. You can create a c...
Author: Ava · Last updated Jul 15, 2026
The marketing team at your organization provides regular updates of a segment of your customer dataset. The marketing team has given you a CSV with 1 million records that must be updated in BigQuery. When you u...
To address the quotaExceeded error you encountered when running the UPDATE statement in BigQuery, let's analyze the options based on the limits of BigQuery's DML (Data Manipulation Language) and the most efficient way to process your data.
A) Reduce the number of records updated each day to stay within the BigQuery UPDATE DML statement limit.
Pros:
- Simple solution: If the issue is due to exceeding BigQuery's DML limits (e.g., the maximum number of rows updated in one statement), reducing the volume of records in each update job can keep you within those limits.
Cons:
- Manual effort required: This approach would involve managing how records are updated in smaller batches, which can be cumbersome, especially if your dataset is large and growing. Additionally, if the dataset grows over time, this method may become increasingly difficult to manage.
- Not a long-term solution: This approach may not be scalable if the volume of records to update increases over time.
When to use: This could work if the dataset update is small and can easily be divided into smaller updates. However, for a dataset with frequent updates (like 1 million records), this approach is not ideal as it would require constant manual intervention.
B) Increase the BigQuery UPDATE DML statement limit in the Quota management section of the Google Cloud Platform Console.
Pros:
- Increased flexibility: Increasing the DML statement limit would theoretically allow you to update more records in one go without hitting the quota exceeded error.
Cons:
- Limits cannot be increased: BigQuery has set DML limits that are not adjustable through the Google Cloud Console or the API. This means you cannot simply increase the update limit to accommodate larger updates.
- Not a feasible solution: Since the quota cannot be changed, this option is not applicable.
When to use: This option is not feasible, as BigQuery doesn't allow you to modify the DML limits for updates.
C) Split the source CSV file into smaller CSV files in Cloud Storage to reduce the number of BigQuery UPDATE DML statements per BigQuery job.
Pros:
- Breaks data into smaller chunks: Splitting the source file into smaller files could help reduce the size of each update operation and potentially avoid the quotaExceeded error.
Cons:
- Complicated m...
Author: Zara · Last updated Jul 15, 2026
As your organization expands its usage of GCP, many teams have started to create their own projects. Projects are further multiplied to accommodate different stages of deployments and target audiences. Each project requires unique access control configurations. The central IT team needs to have access to all projects.
Furthermore, data from Cloud Storage buckets and BigQuery datasets must be shared f...
To simplify access control management in GCP while minimizing the number of policies, you should focus on streamlining and centralizing the management of permissions. Here’s an analysis of the options:
Option A: Use Cloud Deployment Manager to automate access provision.
- Rejected: Cloud Deployment Manager is a tool that helps automate the creation and management of cloud resources. While it can be useful for provisioning infrastructure and services, it doesn’t directly address access control management. Access control is better managed through IAM policies, roles, and resource hierarchy, not through provisioning automation. So, this option doesn't directly address the problem of simplifying access control.
Option B: Introduce resource hierarchy to leverage access control policy inheritance.
- Selected: GCP’s resource hierarchy (organization, folders, projects, etc.) allows policies set at higher levels (like the organization or folder) to be inherited by lower levels (projects). This is a very effective way to simplify access control, because IAM policies applied at the organization or folder level automatically propagate to all contained projects. This helps reduce the need to manage separate policies for each individual project, making the process more streamlined and scalable. This would be a key way to minimize the number of policies and manage access efficiently.
Option C: Create distinct groups for various teams, and specify groups in Cloud IAM policies.
- Selected: This option helps to reduce complexity by creating IAM policies based on groups rather than managing individual user permissions. Using groups simplifies access control because you can manage permissions at the group level instead of managing each team member individually. This...
Author: MoonlitPantherX · Last updated Jul 15, 2026
Your United States-based company has created an application for assessing and responding to user actions. The primary table's data volume grows by 250,000 records per second. Many third parties use your application's APIs to build the functionality into their own frontend applications. Your application's APIs should comply with the f...
Given the scenario, the best option will provide a scalable, globally consistent, and SQL-compliant database solution that meets the real-time data access requirements. Let's go through each option:
Option A: Implement BigQuery with no region selected for storage or processing.
- Rejected: BigQuery is a fully managed, highly scalable data warehouse optimized for analytical workloads and can handle large amounts of data. However, BigQuery is not ideal for transactional systems where you need real-time access to up-to-date data. It focuses on batch processing and reporting, which would not meet the requirements of consistently updating data in real time for API-driven applications. Moreover, BigQuery does not fully support ANSI SQL in the way relational databases like Cloud SQL and Cloud Spanner do, especially for complex transactional workloads.
Option B: Implement Cloud Spanner with the leader in North America and read-only replicas in Asia and Europe.
- Selected: Cloud Spanner is a highly scalable, globally distributed relational database that supports ANSI SQL, and it is specifically designed for high availability and consistency across global applications. It is ideal for high throughput (250,000 records per second) with low-latency global access. Cloud Spanner provides automatic replication across regions and strong consistency across all replicas. This ensures that your application will always have access to the most up-to-date data, regardless of the user's location. By setting the leader in North America and creating read-only replicas in Asia and Europe, you optimize for low-latency access across continents while maintaining data consistency and availability.
Option C: Implement Cloud SQL for PostgreSQL with the master in North America and read replicas in Asia and Europe.
- Rejected: Cloud SQL is a fully mana...
Author: Deepak · Last updated Jul 15, 2026
A data scientist has created a BigQuery ML model and asks you to create an ML pipeline to serve predictions. You have a REST API application with the requirement to serve predictions for an individual user ID with latency under 100 milliseconds. You use the following query to generate predictions: SE...
To create an effective ML pipeline that meets the requirements of serving individual predictions with low latency (under 100 milliseconds), let's analyze each option based on the query and requirements:
Option A: Add a WHERE clause to the query, and grant the BigQuery Data Viewer role to the application service account.
- Rejected: This option suggests using the query directly with a `WHERE` clause for individual user IDs. While it could work for serving predictions on a per-user basis, directly querying BigQuery for predictions in real-time (especially for low-latency requirements) is not optimal. BigQuery is optimized for large-scale data processing and batch queries, but it might not deliver the low-latency response needed for serving predictions with under 100 milliseconds, especially when the query involves complex operations like loading a model and predicting based on features.
Option B: Create an Authorized View with the provided query. Share the dataset that contains the view with the application service account.
- Rejected: An Authorized View can help with access control and simplify queries by hiding sensitive data or creating reusable logic. However, it still relies on querying BigQuery directly for predictions. This would not meet the low-latency requirement, as BigQuery would still be queried for each individual user prediction, which may not respond quickly enough under 100 milliseconds for real-time prediction serving. The view does not inherently solve the latency issue and can still suffer from the same drawbacks as Option A.
Option C: Create a Dataflow pipeline using BigQueryIO to read results from the query. Grant the Dataflow Worker role to the application service account.
- Rejected: Using Dataflow to execute the query and then process results from BigQue...
Author: Aarav2020 · Last updated Jul 15, 2026
You are building an application to share financial market data with consumers, who will receive data feeds. Data is collected from the markets in real time.
Consumers will receive the data in the following ways:
* Real-time event stream
* ANSI SQL...
To provide a solution that handles real-time event streams, ANSI SQL access to real-time and historical data, and batch historical exports, we need a combination of services that support both real-time streaming, batch processing, and SQL querying for data. Let’s analyze each option to identify the best fit.
Option A: Cloud Dataflow, Cloud SQL, Cloud Spanner
- Rejected: While Cloud Dataflow is excellent for real-time data processing, Cloud SQL and Cloud Spanner are both relational databases. Cloud SQL is not designed for high-throughput real-time event streams and might not scale to handle large amounts of streaming market data. Cloud Spanner is better for high-scale transactional data, but it might not be ideal for handling real-time event streams and batch exports efficiently when compared to other solutions designed for big data processing. Additionally, Cloud Spanner might not provide the ideal integration for real-time event stream processing when compared to Cloud Pub/Sub.
Option B: Cloud Pub/Sub, Cloud Storage, BigQuery
- Selected: This option is a strong fit because:
- Cloud Pub/Sub handles the real-time event stream ingestion. It allows you to capture market data as it comes in and transmit it to various consumers in real time.
- Cloud Storage can be used for batch historical exports. You can periodically store market data snapshots in Cloud Storage and make them available for batch processing.
- BigQuery provides ANSI SQL support and allows querying both real-time data (as it is ingested into the system) and historical data (as stored in BigQuery). It can handle large datasets and provide fast querying, both in real-time and batch modes.
This combination ensures that the application can stream da...
Author: Leo · Last updated Jul 15, 2026
You are building a new application that you need to collect data from in a scalable way. Data arrives continuously from the application throughout the day, and you expect to generate approximately 150 GB of JSON data per day by the end of the year. Your requirements are:
* Decoupling producer from consumer
* Space and cost-efficient storage of the raw ingested data, which is to be stored indefinit...
To meet the requirements of your data pipeline (scalable data collection, decoupling of producer and consumer, cost-efficient storage, and real-time SQL querying), let's analyze each option:
Option A: Create an application that provides an API. Write a tool to poll the API and write data to Cloud Storage as gzipped JSON files.
- Rejected: While Cloud Storage is cost-efficient for storing raw data, this option doesn't offer a clean decoupling between the producer and consumer. The application polling the API introduces complexity and potential bottlenecks. Additionally, querying gzipped JSON files directly would not allow for efficient SQL querying. BigQuery and Cloud Storage don't work well together for near real-time SQL querying, which is required in this scenario. It's also not well-suited for transforming data in a way that is efficient for querying or for handling continuous ingestion at scale.
Option B: Create an application that writes to a Cloud SQL database to store the data. Set up periodic exports of the database to write to Cloud Storage and load into BigQuery.
- Rejected: While Cloud SQL can store the data and export it periodically to BigQuery for querying, this option does not scale as well for continuously ingesting large amounts of data, such as the 150 GB of JSON data per day. Cloud SQL is not designed for the kind of high-volume, near-real-time ingestion required here. Also, managing periodic exports can add unnecessary overhead, and querying Cloud SQL directly for large data sets would be inefficient. Furthermore, Cloud SQL is less space-efficient than other storage solutions like Cloud Storage or BigQuery for raw data storage.
Option C: Create an application that publishes events to Cloud Pub/Sub, and create Spark jobs on Cloud Dataproc to convert the JSON data to Avro format, stored on HDFS on Persistent Disk.
- Rejected: While this option decouples the producer and consumer using Cloud Pub/Sub and offers Spark-based transformations, HDFS on Persistent Disk is not the ideal storage solution for cost-efficient storage of raw data that needs to be queried with SQL. HDFS is typically used for large-scale distributed file ...
Author: Sofia · Last updated Jul 15, 2026
You are running a pipeline in Dataflow that receives messages from a Pub/Sub topic and writes the results to a BigQuery dataset in the EU. Currently, your pipeline is located in europe-west4 and has a maximum of 3 workers, instance type n1-standard-1. You notice that during peak periods, your pipeline is struggling to process records in a time...
To address the issue of your pipeline struggling to process records during peak periods, you can consider two main options:
1. Increase the number of max workers (Option A)
- Why this works: Scaling the number of workers can significantly improve the throughput of the pipeline by allowing it to process more records concurrently. When all 3 workers are at maximum CPU utilization, adding more workers will provide the system with additional processing power, reducing backlogs during peak periods.
- Considerations: This solution would work well since the problem arises from insufficient compute capacity. By adding more workers, you allow for greater parallel processing of the incoming data stream.
2. Use a larger instance type for your Dataflow workers (Option B)
- Why this works: By upgrading to a more powerful instance type, such as n1-standard-4 or n1-highmem instances, you can provide more CPU and memory resources to each worker. This can help to reduce the CPU bottleneck, especially if the processing requires more memory or compute power than the current n1-standard-1 instance can provide.
- Considerations: This is a viable option if the performance bottleneck is related to individual workers' CPU or memory limitations. This can provide immediate relief without needing to scale out the number of workers.
Rejected Optio...
Author: Ethan · Last updated Jul 15, 2026
You have a data pipeline with a Dataflow job that aggregates and writes time series metrics to Bigtable. You notice that data is slow to update in Bigtable. This data feeds a dashboard used by thousands of users across the organization. You need to support additional concu...
To address the issue of slow data updates in Bigtable and support additional concurrent users, you need to improve the throughput and efficiency of your pipeline. Let’s review the available options:
1. Increase the maximum number of Dataflow workers by setting maxNumWorkers in PipelineOptions (Option B)
- Why this works: Increasing the number of workers allows the pipeline to scale horizontally and handle more concurrent processing tasks. If the pipeline is underutilized or experiencing delays due to insufficient workers, scaling out will help process more data in parallel, speeding up the write operations to Bigtable. Since you need to support additional users and reduce the time required to write data, this is a direct way to enhance the pipeline’s capacity and efficiency.
- Considerations: If your pipeline is bottlenecked by the number of workers (too few workers to handle high throughput), increasing the workers will address the bottleneck.
2. Increase the number of nodes in the Bigtable cluster (Option C)
- Why this works: Bigtable’s performance is highly dependent on the number of nodes in the cluster. By increasing the number of nodes, you expand the processing and storage capacity of Bigtable, which can improve the time it takes to write and update data. Since you are handling time-series metrics with thousands of users, ensuring Bigtable can handle large-scale concurrent writes and reads is essential.
- Considerations: If Bigtable is overwhelmed by the number of writes or the data size, scaling the Bigtable cluster will provide the necessary resources to keep up with the load and reduce delays in data up...
Author: Ravi Patel · Last updated Jul 15, 2026
You have several Spark jobs that run on a Cloud Dataproc cluster on a schedule. Some of the jobs run in sequence, and some of the jobs run concu...
To automate the scheduling and execution of Spark jobs on a Cloud Dataproc cluster, it's important to choose a solution that ensures proper job sequencing and concurrency while integrating with Cloud Dataproc efficiently. Let's review each option:
1. Create a Cloud Dataproc Workflow Template (Option A)
- Why this works: Cloud Dataproc Workflow Templates provide an ideal way to automate and schedule Spark jobs on a cluster. These templates allow you to define a series of jobs to run in sequence or in parallel, with support for job dependencies. This option enables you to automate the execution flow of Spark jobs with clear control over which jobs run in parallel or sequentially.
- Considerations: This solution is tightly integrated with Cloud Dataproc, providing an efficient way to manage and automate workflows for scheduled jobs. It eliminates the need for external orchestration tools and handles job sequencing and concurrency out-of-the-box.
2. Create an initialization action to execute the jobs (Option B)
- Why this is rejected: Initialization actions in Cloud Dataproc are used to customize the environment or install additional software on a cluster when it is created. However, initialization actions are not designed to automate job execution. They execute before the cluster is fully operational and are more suited for setup tasks, not for orchestrating job execution. Using initialization actions for job execution would be an improper use case, leading to unnecessary complexity.
3. Create a Directed Acyclic Gr...
Author: StarryEagle42 · Last updated Jul 15, 2026
You are building a new data pipeline to share data between two different types of applications: jobs generators and job runners. Your solution must scale to accommodate increases in usage and must accommodate the addition of n...
To choose the best solution for building a scalable data pipeline that facilitates communication between job generators and job runners, it’s important to consider the scalability, flexibility, and ease of integration of each option. Let’s analyze the available options:
1. Use a Cloud Pub/Sub topic to publish jobs, and use subscriptions to execute them (Option B)
- Why this works: Cloud Pub/Sub is a fully managed messaging service designed for scalable and real-time messaging. It allows decoupling of the job generators (publishers) and job runners (subscribers). With Pub/Sub, you can easily scale to accommodate increases in usage because it can handle large volumes of messages and automatically scale to meet demand. Furthermore, you can add new applications (subscribers) without affecting existing ones by simply subscribing to the relevant Pub/Sub topics. This option allows for asynchronous communication, high availability, and flexibility.
- Considerations: This solution fits perfectly for scenarios where different applications (job generators and job runners) need to interact with each other asynchronously. Pub/Sub is designed for high scalability and allows seamless addition of new subscribers (applications) without disrupting existing systems.
2. Create an API using App Engine to receive and send messages to the applications (Option A)
- Why this is rejected: While App Engine can certainly serve as an API platform for sending and receiving messages, it is more appropriate for building web services and applications rather than acting as a messaging layer between job ...
Author: Stella · Last updated Jul 15, 2026
You need to create a new transaction table in Cloud Spanner that stores product sales data. You are deciding what to use as a primary key. Fro...
When choosing a primary key for a transaction table in Cloud Spanner, you need to consider performance factors such as how efficiently the database can handle inserts, queries, and how the data is distributed across nodes. Let's examine each option:
1. The current epoch time (Option A)
- Why this is rejected: While using epoch time as a primary key provides unique values, it is not optimal for Cloud Spanner, especially for high-insert throughput. If transactions are generated in rapid succession, the keys may not be distributed well across Cloud Spanner nodes, which can result in hotspots and poor performance due to localized contention on specific partitions. This would reduce scalability and increase latency for inserts and queries.
2. A concatenation of the product name and the current epoch time (Option B)
- Why this is rejected: Concatenating the product name and epoch time introduces a combination that can be unique but may not provide optimal distribution of keys. The product name may create skewed distribution if a few product names are dominant, leading to hotspots, which would hurt scalability. Additionally, the epoch time, while time-based, can still lead to sequential patterns and clustering of data in a small range, causing poor distribution.
3. A random universally unique iden...
Author: Noah · Last updated Jul 15, 2026
Data Analysts in your company have the Cloud IAM Owner role assigned to them in their projects to allow them to work with multiple GCP products in their projects. Your organization requires that all BigQuery data access logs be retained for 6 months. You need to ens...
To address the requirement of ensuring that only audit personnel can access the data access logs for all projects while retaining them for 6 months, let’s evaluate the options one by one:
A) Enable data access logs in each Data Analyst's project. Restrict access to Stackdriver Logging via Cloud IAM roles.
- Analysis: This option involves enabling data access logs directly in each Data Analyst's project. While restricting access to Stackdriver Logging via Cloud IAM roles may control access, it is not efficient. Different teams may manage IAM permissions for individual projects, which can lead to complex and hard-to-manage configurations. The data access logs are not centralized in this setup, and the long-term retention requirement (6 months) may be challenging to enforce consistently across projects.
- Rejected: The decentralized management of data logs and difficulty in ensuring consistent retention across projects make this option less practical.
B) Export the data access logs via a project-level export sink to a Cloud Storage bucket in the Data Analysts' projects. Restrict access to the Cloud Storage bucket.
- Analysis: This option exports logs to Cloud Storage within the Data Analysts' own projects. While this gives the flexibility of project-level configuration, it creates a problem with centralized access control. The audit personnel would need to manage access to potentially many Cloud Storage buckets across different projects. Additionally, long-term retention management for logs across multiple projects could become cumbersome.
- Rejected: The decentralized management and complexity of en...
Author: Ava · Last updated Jul 15, 2026
Each analytics team in your organization is running BigQuery jobs in their own projects. You want to enable each team to monitor...
Let's evaluate each option to determine the best solution for enabling each analytics team to monitor slot usage within their own projects.
A) Create a Cloud Monitoring dashboard based on the BigQuery metric `query/scanned_bytes`
- Analysis: This option tracks the `scanned_bytes` metric, which measures the amount of data processed by BigQuery queries. While this is useful for understanding the data volume being handled, it does not directly track slot usage, which is what the teams need for monitoring BigQuery performance.
- Rejected: This does not address the core need of tracking slot usage, making it an inappropriate option.
B) Create a Cloud Monitoring dashboard based on the BigQuery metric `slots/allocated_for_project`
- Analysis: The metric `slots/allocated_for_project` provides information on the number of BigQuery slots allocated to a particular project, which is more relevant to monitoring slot usage. However, this metric is related to the overall allocation, not the actual usage of slots. The analytics teams need to monitor how efficiently those slots are being used, not just how many are allocated to them.
- Rejected: While this is closer to the target, it doesn't track actual slot usage or performance, making it insufficient for detailed monitoring.
C) Create a log export for each project, capture the BigQuery job execution logs, create a custom metric based on the `totalSlotMs`, and create a Cloud Monitoring dashboard based on the custom metric
- Analysis: Th...
Author: Leah Davis · Last updated Jul 15, 2026
You are operating a streaming Cloud Dataflow pipeline. Your engineers have a new version of the pipeline with a different windowing algorithm and triggering strategy. You want to update the running pipeline with th...
Let's evaluate each option to determine the best approach for updating a running streaming Cloud Dataflow pipeline while ensuring that no data is lost.
A) Update the Cloud Dataflow pipeline inflight by passing the `--update` option with the `--jobName` set to the existing job name
- Analysis: The `--update` option can be used to update a running Cloud Dataflow pipeline with a new version of the code. This option updates the job while it is still running, and by specifying the existing job name, it ensures that the pipeline continues to run without losing data. However, this approach requires the new version of the pipeline to be compatible with the existing windowing and triggering strategies to ensure smooth operation.
- Selected: This option works well because it enables an in-place update of the pipeline without having to stop it. It ensures no data loss by maintaining the same job name and allowing the pipeline to keep running while the new code is applied.
B) Update the Cloud Dataflow pipeline inflight by passing the `--update` option with the `--jobName` set to a new unique job name
- Analysis: This approach would create a new job with a different name, even though the `--update` option is used. This is not appropriate for maintaining a running pipeline without data loss because creating a new job will cause the pipeline to restart, resulting in a temporary interruption and possible data loss.
- Rejected: This option is not ideal as it leads to the creation...
Author: Chloe · Last updated Jul 15, 2026
You need to move 2 PB of historical data from an on-premises storage appliance to Cloud Storage within six months, and your outbound network capacity is constra...
Let's evaluate the best approach for migrating 2 PB of historical data from an on-premises storage appliance to Cloud Storage within the specified time frame of six months, with the constraint of 20 Mb/sec outbound network capacity.
A) Use Transfer Appliance to copy the data to Cloud Storage
- Analysis: The Transfer Appliance is a physical device provided by Google Cloud that allows you to transfer large amounts of data to Cloud Storage. This option is ideal when dealing with large datasets and limited network bandwidth, as it enables you to physically load the data onto the appliance and then ship it to a Google data center, where it will be uploaded to Cloud Storage. The Transfer Appliance is designed for situations like this, where network bandwidth is a limiting factor.
- Selected: This is the best option because it bypasses the network constraints and enables efficient, high-speed data transfer by leveraging physical shipping. It's designed for large-scale migrations like this, where network throughput is not sufficient to handle the data volume within the given time frame.
B) Use gsutil cp to compress the content being uploaded to Cloud Storage
- Analysis: While compression can help reduce the amount of data that needs to be transferred, it will not overcome the main bottleneck in this case: the 20 Mb/sec outbound network capacity. Even with compression, it would still take a significant amount of time to transfer 2 PB of data, which is unrealistic within the six-month period. Additionally, compressing large datasets can be resource-intensive and may not drastically reduce the transfer time.
- Rejected: This option won't address the network bandwidth limitation effectively and may still take too long fo...
Author: Aditya · Last updated Jul 15, 2026
You receive data files in CSV format monthly from a third party. You need to cleanse this data, but every third month the schema of the files changes. Your requirements for implementing these transformations include:
* Executing the transformations on a schedule
* Enabling non-dev...
Let's evaluate each option in terms of the requirements: executing transformations on a schedule, enabling non-developer analysts to modify transformations, and providing a graphical tool for designing transformations.
A) Use Dataprep by Trifacta to build and maintain the transformation recipes, and execute them on a scheduled basis
- Analysis: Dataprep by Trifacta is specifically designed for data cleansing and transformation tasks. It provides a user-friendly, graphical interface that allows non-developer analysts to design and modify transformations without writing code. Additionally, Dataprep allows you to schedule transformations, making it well-suited for monthly data ingestion with schema changes every third month. You can also build and maintain transformation recipes, which makes it easy to adapt to schema changes over time.
- Selected: This option meets all the requirements, especially the need for a graphical tool, the ability to enable non-developers to modify transformations, and scheduling functionality. It provides a highly suitable and scalable solution for the scenario described.
B) Load each month's CSV data into BigQuery, and write a SQL query to transform the data to a standard schema. Merge the transformed tables together with a SQL query
- Analysis: While loading data into BigQuery and using SQL for transformation is a common approach, it doesn’t provide a graphical interface for non-developers to modify transformations. Analysts would need to write or modify SQL queries to adapt to schema changes, which requires more technical knowledge. Additionally, this approach does not offer an easy way to schedule the transformations and maintain them over time.
- Rejected: This approach is more suitable for developers or analysts with SQL skills. It lacks the graphical interface that would allow non-developers to easily modify transformations, and scheduling transformations would require additional tools like Cloud Scheduler or BigQuery scheduled queries,...
Author: Oscar · Last updated Jul 15, 2026
You want to migrate an on-premises Hadoop system to Cloud Dataproc. Hive is the primary tool in use, and the data format is Optimized Row Columnar (ORC).
All ORC files have been successfully copied to a Cloud Storage bucket. You need to replicate some data to the cluster's local Hadoop Di...
Let's break down each option and analyze them based on the context of migrating an on-premises Hadoop system to Cloud Dataproc, where the primary tool in use is Hive, and the data format is Optimized Row Columnar (ORC):
A) Run the gsutil utility to transfer all ORC files from the Cloud Storage bucket to HDFS. Mount the Hive tables locally.
- Analysis: This option requires copying all ORC files from Cloud Storage to HDFS using `gsutil`, which could involve significant data transfer costs and time, especially for large datasets. Also, managing the replication process could become cumbersome for dynamic data. Mounting the Hive tables locally on HDFS is not efficient, as it does not fully leverage the cloud-native capabilities of Dataproc (such as Cloud Storage integration).
- Rejected because: It does not take advantage of Cloud Storage, which is designed to be directly accessible from Dataproc, leading to inefficient storage and data handling. It might also introduce unnecessary complexity and cost with the need to replicate data on HDFS.
B) Run the gsutil utility to transfer all ORC files from the Cloud Storage bucket to any node of the Dataproc cluster. Mount the Hive tables locally.
- Analysis: This option involves copying the ORC files to an individual Dataproc node. However, mounting Hive tables locally could cause performance bottlenecks since the files will not be accessible directly from all cluster nodes. Data replication and management could also become inefficient as you'd have to manage the files on one node instead of leveraging cloud-native features for distributed access.
- Rejected because: It restricts the data's accessibility to a single node, and Hive performance could suffer from potential I/O bottlenecks and scaling issues.
C) Run the gsutil utility to transfer all ORC files from the Cloud Storage bucket to the master node of the Dataproc cluster. Then run the Hadoop utility to copy them to HDFS. Mount the Hive tables from HDFS.
- Analysis: This option involves transferring files from Cloud Storage to the master node, then using Hadoop commands to copy the...
Author: Rahul · Last updated Jul 15, 2026
You are implementing several batch jobs that must be executed on a schedule. These jobs have many interdependent steps that must be executed in a specific order. Portions of the jobs involve executing shell scripts, running Hadoop jobs, and running queries in BigQuery. The jobs are expected to run for many minutes up to severa...
To determine the best service for managing the execution of these interdependent batch jobs with retries, we need to carefully analyze each option based on their capabilities and suitability for the task.
A) Cloud Scheduler
- Analysis: Cloud Scheduler is a fully managed cron job service that allows you to trigger HTTP requests, Pub/Sub messages, or Cloud Functions at scheduled times. While Cloud Scheduler can be used to initiate jobs at specific intervals, it doesn’t manage complex workflows or interdependent steps. It is more suited for simple cron-style scheduling tasks rather than managing multi-step workflows with retries and interdependencies.
- Rejected because: It cannot manage the complex workflow with multiple interdependent steps or the retry logic as required by the scenario. It only triggers tasks based on a schedule and does not provide built-in orchestration.
B) Cloud Dataflow
- Analysis: Cloud Dataflow is primarily designed for processing large amounts of data in real-time or batch mode. It is ideal for data pipelines and transformations, particularly when working with data from sources like Pub/Sub, Cloud Storage, or BigQuery. However, while it supports data processing jobs and fault tolerance, it is not a workflow orchestration tool for managing non-data-specific tasks (like executing shell scripts or running Hadoop jobs) and interdependent job steps.
- Rejected because: It is not suited for orchestrating a variety of tasks outside its data processing pipeline context, such as managing shell scripts or interdependent system-level operations. It also doesn't provide the full flexibility of retry mechanisms for a variety of different job types.
C) Cloud Functions
- Analysis: Cloud Functions allows you to run small, event-driven functions in the cloud. These are best suited for lightweight, stateless jobs triggered by events, such as chang...
Author: Emma · Last updated Jul 15, 2026
You work for a shipping company that has distribution centers where packages move on delivery lines to route them properly. The company wants to add cameras to the delivery lines to detect and track any visual damage to the packages in transit. You need to create a way to automate the detection of ...
To determine the best solution for automating the detection of damaged packages in transit, we need to consider factors like real-time processing, ease of integration, scalability, and the ability to perform image analysis. Let’s evaluate each option:
A) Use BigQuery machine learning to be able to train the model at scale, so you can analyze the packages in batches.
- Analysis: BigQuery ML is designed for machine learning tasks on structured data (e.g., tabular data) and is ideal for analyzing large datasets. However, it is not typically used for processing unstructured data like images in real-time. BigQuery ML works well for batch processing, not for image-based analysis of package damage during transit.
- Rejected because: It is not suitable for real-time image analysis. BigQuery ML operates on structured data and would not be efficient for detecting damage in images while packages are in transit.
B) Train an AutoML model on your corpus of images, and build an API around that model to integrate with the package tracking applications.
- Analysis: Google Cloud AutoML allows for the easy creation of custom machine learning models, especially for tasks like image classification, which is exactly what we need for detecting damage in images. By training the model on a corpus of images of damaged and undamaged packages, you can create a model that can automatically detect damage in real-time. Additionally, wrapping the model in an API makes it easy to integrate with existing tracking applications.
- Selected because: This option provides a scalable, real-time solution for detecting damaged packages by leveraging AutoML for custom image classification. It integrates easily with package tracking applications, ensuring that damaged packages are flagged for review in real time.
C) Use the Cloud Vision API to detect for damage, and raise an alert through Cloud Functions. Integrate the package tracking applications with this function.
- Analysis: The Cloud Vision API can analyze images...
Author: Chloe · Last updated Jul 15, 2026
You are migrating your data warehouse to BigQuery. You have migrated all of your data into tables in a dataset. Multiple users from your organization will be using the data. They should only see...
To ensure that users can only see specific tables based on their team membership, we need a solution that provides secure, granular access control to data in BigQuery. Let’s evaluate each option based on the requirements.
A) Assign the users/groups data viewer access at the table level for each table
- Analysis: This option involves giving users or groups permission to access specific tables directly. While this approach is straightforward, it can quickly become difficult to manage as the number of tables and users grows. You would need to assign and manage permissions individually for each table, leading to a complex and error-prone process.
- Rejected because: It’s not scalable or efficient for managing permissions across multiple users and tables. It would require constant maintenance as users change or new tables are added.
B) Create SQL views for each team in the same dataset in which the data resides, and assign the users/groups data viewer access to the SQL views
- Analysis: This option involves creating SQL views for each team and giving the team access to those views. While this can simplify access management, SQL views do not provide the same level of security as authorized views in BigQuery. Users would still be able to query the underlying tables if they have sufficient privileges, potentially allowing them to access restricted data.
- Rejected because: SQL views can be easily bypassed if users have access to the underlying tables. They don't provide the fine-grained access control that authorized views offer.
C) Create authorized views for each team in the same dataset in which the data resides, and assign the users/groups data viewer access to the authorized views
- Analysis: This approach involves creating authorized views for each team, which can restrict access to only certain tables and data within a dataset. Authorized views are a more secure way of managing acces...
Author: Sophia Clark · Last updated Jul 15, 2026
You want to build a managed Hadoop system as your data lake. The data transformation process is composed of a series of Hadoop jobs executed in sequence.
To accomplish the design of separating storage from compute, you decided to use the Cloud Storage connector to store all input data, output data, and intermediary data. However, you noticed that one Hadoop job runs very slowly with Cloud Dataproc, when compared with the on-premi...
To resolve the performance issue for the disk I/O intensive Hadoop job on Cloud Dataproc, we need to focus on addressing the root cause of the slowdown, which is disk I/O. Let's evaluate the given options:
A) Allocate sufficient memory to the Hadoop cluster, so that the intermediary data of that particular Hadoop job can be held in memory
- Analysis: This option is focused on increasing memory to hold intermediary data in memory. While more memory can help in certain scenarios, it may not solve the issue if the job is heavily disk I/O bound, meaning it relies on reading and writing data to disk frequently. Even if memory is increased, the underlying bottleneck of disk I/O will still exist, and the job will continue to suffer from slow performance.
- Rejected because: This approach doesn't address the core problem of disk I/O. The Hadoop job is still dependent on disk operations, and increasing memory may not provide significant improvements in this case.
B) Allocate sufficient persistent disk space to the Hadoop cluster, and store the intermediate data of that particular Hadoop job on native HDFS
- Analysis: This option suggests using native HDFS, which is optimized for handling large-scale distributed data processing. Cloud Dataproc’s HDFS runs on local disk storage attached to the cluster nodes, and it is better suited for performance with disk I/O intensive jobs compared to Cloud Storage, which introduces additional latency. By moving the intermediary data from Cloud Storage to local HDFS, the job can benefit from faster disk access, improving the overall performance.
- Selected because: This option directly addresses the disk I/O performance issue by moving intermediary data to native HDFS, which is optimized for high throughput and low latency access to data. Using HDFS instead of Cloud Storage will speed up the disk operations significantly.
C) Allocate more CPU cores of the virtual machine instances of the Hadoop...
Author: Grace · Last updated Jul 15, 2026
You work for an advertising company, and you've developed a Spark ML model to predict click-through rates at advertisement blocks. You've been developing everything at your on-premises data center, and now your company is migrating to Google Cloud. Your data center will be closing soon, so a rapid lift-and-shift migration is necessary. However, the data you've been using will be mig...
In the given scenario, the company is migrating from an on-premises data center to Google Cloud, and the primary goal is to quickly move the existing Spark ML training pipelines while integrating with BigQuery for data access. Let’s analyze each option and see which best meets the company's needs.
Option A: Use Vertex AI for training existing Spark ML models
- Analysis: Vertex AI is a managed service on Google Cloud for machine learning tasks, but it is primarily designed for TensorFlow, PyTorch, and other similar frameworks. It does support Spark ML through the integration with DataFlow and Apache Beam, but Vertex AI is generally better suited for models that are built specifically within the Google Cloud ecosystem (e.g., TensorFlow or other deep learning models).
- Rejection Reason: While Vertex AI is great for many use cases, it is not the most straightforward way to directly run legacy Spark ML models, and it would require significant adaptation of the Spark ML code to integrate it into Vertex AI.
- When to use: If you’re starting fresh with TensorFlow or want to transition from legacy models, Vertex AI might be more suitable.
Option B: Rewrite your models on TensorFlow, and start using Vertex AI
- Analysis: Rewriting the Spark ML models to use TensorFlow would require a significant amount of effort and rework. The models in Spark ML have their own pipelines, libraries, and specific handling that would not directly map to TensorFlow. Additionally, migrating to TensorFlow would involve a time-consuming process and might not provide the best value if Spark ML is working well for the use case.
- Rejection Reason: This approach requires unnecessary rewriting of the models, which defeats the purpose of quickly migrating and retraining existing models. The company is aiming for a lift-and-shift migration, so this would delay the transition unnecessarily.
- When to use: This could be a long-term option if the company wanted to completely transition to TensorFlow, but for a quick mi...
Author: Samuel · Last updated Jul 15, 2026
You work for a global shipping company. You want to train a model on 40 TB of data to predict which ships in each geographic region are likely to cause delivery delays on any given day. The model will be based on multiple attributes collected from multiple sources. Telemetry data, including location in GeoJSON format, will be pulled from each ship and loaded every hour. You want to have a dashboard that shows how many and which ships ...
In the given scenario, we need to select a storage solution that can handle large volumes of data, support geospatial processing, and integrate with prediction models. Let's analyze each option based on these requirements.
Option A: BigQuery
- Analysis: BigQuery is a fully managed, serverless, highly scalable data warehouse solution designed to handle large volumes of data. It supports geospatial processing natively using the `GEOGRAPHY` data type, which allows for efficient querying and analysis of GeoJSON or other geospatial data formats. Additionally, BigQuery integrates well with Google Cloud AI/ML tools for predictive modeling. BigQuery is highly scalable, allowing for the analysis of 40 TB of data without issues. It also integrates with visualization tools (like Google Data Studio or Looker), making it easy to create dashboards.
- Selection Reason: BigQuery is ideal for this scenario because it can easily store and process the large 40 TB dataset, supports geospatial data natively, integrates well with machine learning tools (like Vertex AI), and can be queried efficiently to build the desired dashboard. Additionally, BigQuery’s ability to manage large amounts of data and run complex queries is perfect for handling telemetry data that is updated every hour.
- When to use: BigQuery is best used for large-scale analytics, especially when you need to store vast amounts of data, perform geospatial queries, and integrate with AI/ML models.
Option B: Cloud Bigtable
- Analysis: Cloud Bigtable is a NoSQL, scalable database optimized for real-time analytics on large datasets. It is well-suited for time-series data, such as sensor data, but does not have built-in support for geospatial data or geospatial queries like BigQuery does. While Bigtable can handle large datasets efficiently, it would require significant custom development to perform geospatial analysis and predictions.
- Rejection Reason: Although Cloud Bigtable is excellent for handling time-series data, it does not offer native support for geospatial data and lacks advanced query capabilities that BigQuery provides for geographic and predictive analytics. It also lacks integration with built-in geospatial functions ...
Author: Emma Brown · Last updated Jul 15, 2026
You operate an IoT pipeline built around Apache Kafka that normally receives around 5000 messages per second. You want to use Google Cloud Platform to create an alert as soon as the movi...
In this scenario, the goal is to monitor a stream of data coming from Apache Kafka, calculate the moving average of messages per second over a 1-hour window, and trigger an alert if the average drops below 4000 messages per second. Let's evaluate each option based on its ability to handle the required tasks effectively.
Option A: Consume the stream of data in Dataflow using Kafka IO. Set a sliding time window of 1 hour every 5 minutes. Compute the average when the window closes, and send an alert if the average is less than 4000 messages.
- Analysis: This approach leverages Google Cloud Dataflow, which is well-suited for stream processing. A sliding window every 5 minutes will allow for continuous monitoring of the data stream and computation of the moving average. This option is very efficient for real-time data processing, as it will compute the moving average dynamically in smaller time intervals (e.g., every 5 minutes) and trigger alerts as soon as the threshold condition is met.
- Selection Reason: This option is ideal because it allows for real-time monitoring of the message stream with a moving average calculation on a sliding window. Dataflow is highly scalable and can process large volumes of streaming data in real-time, making it an efficient solution for the task at hand.
- When to use: Use this option when you need real-time stream processing and are aiming to trigger alerts dynamically based on moving averages or other complex conditions.
Option B: Consume the stream of data in Dataflow using Kafka IO. Set a fixed time window of 1 hour. Compute the average when the window closes, and send an alert if the average is less than 4000 messages.
- Analysis: This option uses a fixed time window of 1 hour to compute the average, which may not provide immediate feedback when the message rate drops below 4000 messages per second. The alert will only be triggered after the full 1-hour window has passed and the average is computed. This approach does not provide a moving average over time and could lead to delays in identifying issues with the message stream.
- Rejection Reason: The fixed time window might introduce latency because it only checks the average once every hour. Since the requirement is to detect a drop in the message rate as soon as it happens, a fixed window could cause unnecessary delays in triggering alerts.
- When to use: This option is suitable for scenarios where checking the data in hourly intervals is acceptable and immediate responsiveness is not critical.
Option C...
Author: Ella · Last updated Jul 15, 2026
You plan to deploy Cloud SQL using MySQL. You need to ensure high availability in the event of a zon...
When deploying Cloud SQL with MySQL, it's essential to ensure high availability (HA) in case of a zone failure. High availability involves minimizing downtime and ensuring that there is a backup system that can take over if the primary instance becomes unavailable. Let's analyze the options based on the requirement for high availability and zone failure recovery.
Option A: Create a Cloud SQL instance in one zone, and create a failover replica in another zone within the same region.
- Analysis: This option involves setting up a high availability (HA) configuration with Cloud SQL, where the primary instance is in one zone, and a failover replica is deployed in another zone within the same region. The failover replica is a synchronous copy of the primary instance, ensuring that data is always up-to-date. If the primary instance becomes unavailable due to a zone failure, the system will automatically fail over to the replica. This is the recommended and supported approach to ensuring high availability within a region.
- Selection Reason: This option provides true high availability with minimal downtime because it ensures that there is a replica instance ready to take over in case of a zone failure. It's the best solution for ensuring high availability during zone failures.
- When to use: This option is ideal when you need automatic failover and minimal downtime in the event of a zone failure.
Option B: Create a Cloud SQL instance in one zone, and create a read replica in another zone within the same region.
- Analysis: A read replica in a different zone can improve read scalability, but it does not provide automatic failover capabilities. In case the primary instance goes down, a read replica cannot automatically take over as the new primary instance. You would need to manually promote the read replica, leading to potential downtime during the transition.
- Rejection Reason: This option does not provide the automatic failover necessary for high availability. While it can help with scaling reads, it doesn't address the need for recovery in the e...
Author: Ella · Last updated Jul 15, 2026
Your company is selecting a system to centralize data ingestion and delivery. You are considering messaging and data integration systems to address the requirements. The key requirements are:
* The ability to seek to a particular offset in a topic, possibly back to the start of all data ever captured...
Let's evaluate the options based on the key requirements:
Key Requirements:
1. The ability to seek to a particular offset in a topic, possibly back to the start of all data ever captured.
2. Support for publish/subscribe semantics on hundreds of topics.
3. Retain per-key ordering.
Option A: Apache Kafka
- Analysis: Apache Kafka is a highly scalable and distributed messaging system designed to handle large volumes of data streams. It supports seeking to a particular offset within a topic, allowing you to rewind to earlier messages (even to the start of the topic, if necessary). Kafka also natively supports publish/subscribe semantics and can handle hundreds of topics effectively. Additionally, Kafka guarantees per-key ordering through its partitioning mechanism, ensuring that messages with the same key are processed in order within a partition.
- Selection Reason: Apache Kafka is the best option for meeting all of the given requirements. It provides robust support for managing large-scale, real-time data streams with high availability, fault tolerance, and precise control over data offsets. Kafka's ability to retain per-key ordering is crucial for applications that need strict message ordering for each key, and its publish/subscribe model supports large numbers of topics.
- When to use: Kafka is ideal for high-throughput data pipelines, real-time analytics, event streaming, and systems where per-key ordering and offset control are critical.
Option B: Cloud Storage
- Analysis: Cloud Storage is primarily designed for object storage, and it does not natively support seeking to specific offsets or real-time publish/subscribe messaging. It can store large amounts of data, but it lacks the streaming and real-time data processing features necessary for this use case. Additionally, Cloud Storage does not provide the ability to maintain per-key ordering in the context of message streams.
- Rejection Reason: Cloud Storage is not suitable for real-time messaging, seeking to offsets, or supporting publish/subscribe semantics. It is mor...
Author: CrimsonViperX · Last updated Jul 15, 2026
You are planning to migrate your current on-premises Apache Hadoop deployment to the cloud. You need to ensure that the deployment is as fault-tolerant and cost-effective as possible for...
When migrating an on-premises Apache Hadoop deployment to the cloud, the goal is to ensure fault tolerance, cost-effectiveness, and optimized resource usage for long-running batch jobs. Here’s an analysis of each option:
Option A: Deploy a Dataproc cluster. Use a standard persistent disk and 50% preemptible workers. Store data in Cloud Storage, and change references in scripts from hdfs:// to gs://
- Fault Tolerance: Dataproc is a fully managed service that automatically handles cluster management, making it fault-tolerant. It can recover from failures, but using 50% preemptible workers could introduce some risk as preemptible VMs can be terminated at any time.
- Cost-effectiveness: Preemptible workers are cheaper, which is beneficial for cost management, especially for batch processing jobs that can tolerate some interruptions.
- Storage: Using Cloud Storage (gs://) for data is ideal as it’s designed for scalability and cost-effectiveness and can easily replace HDFS in a cloud environment.
- Why Selected: This option strikes a good balance between cost-effectiveness (preemptible VMs) and fault tolerance (Dataproc cluster management). The use of standard persistent disks ensures reliable storage performance.
Option B: Deploy a Dataproc cluster. Use an SSD persistent disk and 50% preemptible workers. Store data in Cloud Storage, and change references in scripts from hdfs:// to gs://
- Fault Tolerance: Similar to Option A, Dataproc offers managed services for fault tolerance. However, the use of SSD disks over standard persistent disks offers better performance but may increase cost.
- Cost-effectiveness: Preemptible workers are cost-effective, but SSD persistent disks increase storage costs compared to standard disks.
- Storage: Cloud Storage remains an optimal choice for cost-effective, scalable storage.
- Why Rejected: While this option improves performance with SSD disks, the additional cost may not justify the...
Author: Scarlett · Last updated Jul 15, 2026
Your team is working on a binary classification problem. You have trained a support vector machine (SVM) classifier with default parameters, and received an area under the Curve (AUC) of 0.87 o...
To improve the AUC of your Support Vector Machine (SVM) model, the goal is to refine the model or its parameters to achieve better generalization and performance. Here's an analysis of each option:
Option A: Perform hyperparameter tuning
- Reasoning: SVM classifiers are highly sensitive to hyperparameters such as the regularization parameter (C) and the kernel function parameters (e.g., gamma for the RBF kernel). By performing hyperparameter tuning (e.g., using grid search or random search), you can optimize these parameters to improve the model’s performance and potentially increase the AUC.
- Why Selected: Hyperparameter tuning is a standard approach to improving SVM performance. The AUC of 0.87 indicates that there’s room for improvement in the model, and adjusting the hyperparameters can lead to better model performance.
- Scenarios: This option is generally applicable in situations where you are already satisfied with the basic model but want to fine-tune its performance.
Option B: Train a classifier with deep neural networks, because neural networks would always beat SVMs
- Reasoning: While deep neural networks (DNNs) have been successful in many tasks, it is incorrect to assume that they would always outperform SVMs. SVMs are effective for binary classification problems, especially in scenarios with a smaller dataset or fewer features. Switching to neural networks introduces more complexity and the risk of overfitting, particularly if you don't have a sufficiently large dataset or computational resources.
- Why Rejected: There's no guarantee that DNNs would always outperform SVMs, especially with s...
Author: Suresh · Last updated Jul 15, 2026
You need to deploy additional dependencies to all nodes of a Cloud Dataproc cluster at startup using an existing initialization action. Company security policies require that Cloud Dataproc nodes do not have access...
When deploying additional dependencies to all nodes in a Cloud Dataproc cluster while adhering to company security policies that prevent internet access, the solution must ensure that resources are available to the nodes without requiring internet access. Let’s break down the options:
Option A: Deploy the Cloud SQL Proxy on the Cloud Dataproc master
- Reasoning: The Cloud SQL Proxy is used to securely connect to Cloud SQL databases from a local environment or a Google Cloud instance. While this could be useful for database access, it does not address the primary issue of ensuring that dependencies are available during startup. The proxy itself doesn't solve the problem of accessing public initialization actions or external resources without internet access.
- Why Rejected: This option is specific to managing database connections and doesn’t solve the broader problem of fetching dependencies during cluster initialization.
Option B: Use an SSH tunnel to give the Cloud Dataproc cluster access to the Internet
- Reasoning: Setting up an SSH tunnel could, in theory, allow the cluster to access external resources through a secure channel. However, this would complicate the setup and may violate company security policies, which prohibit internet access for the Dataproc nodes. Additionally, using an SSH tunnel is not a robust solution for managing dependencies, especially if you need to automate the process across multiple nodes.
- Why Rejected: This solution would likely compromise security policies and is not a scalable or recommended approach for deploying dependencies at scale.
...
Author: Alexander · Last updated Jul 15, 2026
You need to choose a database for a new project that has the following requirements:
* Fully managed
* Able to automatically scale up
* Transactionally consistent
* Able to s...
To select the appropriate database for your new project based on the specified requirements, let's analyze each option:
Requirements:
- Fully managed: The database should be a managed service.
- Able to automatically scale up: The database should be able to scale horizontally or vertically as needed.
- Transactionally consistent: The database should support ACID (Atomicity, Consistency, Isolation, Durability) transactions.
- Able to scale up to 6 TB: The database should support data volumes of up to 6 TB.
- Able to be queried using SQL: The database should support SQL for querying.
Option A: Cloud SQL
- Fully Managed: Cloud SQL is a fully managed relational database service.
- Scalability: Cloud SQL supports vertical scaling (increasing instance size) but doesn't automatically scale horizontally for large data volumes or high throughput beyond a certain point.
- Transactionally Consistent: Cloud SQL is ACID-compliant and supports transaction consistency.
- Scalability to 6 TB: Cloud SQL can handle data volumes up to 64 TB (depending on the database engine), so 6 TB is feasible.
- SQL Support: Cloud SQL supports SQL (MySQL, PostgreSQL, SQL Server).
- Why Rejected: Although Cloud SQL meets most of the requirements, it does not support automatic horizontal scaling as seamlessly as other options (like Cloud Spanner) and could be less efficient for very large-scale applications, especially when automatic scaling is critical.
Option B: Cloud Bigtable
- Fully Managed: Cloud Bigtable is a fully managed NoSQL database designed for large-scale, low-latency workloads.
- Scalability: Cloud Bigtable can scale horizontally and is suitable for workloads that require massive scale.
- Transactionally Consistent: Cloud Bigtable does not support full ACID transactions. It is designed for high throughput and low latency but lacks support for transactional consistency across multiple rows or tables.
- Scalability to 6 TB: Cloud Bigtable easily supports data volumes much g...
Author: Ava · Last updated Jul 15, 2026
You work for a mid-sized enterprise that needs to move its operational system transaction data from an on-premises database to GCP. The databa...
When choosing a database to move a 20 TB operational system transaction data to Google Cloud Platform (GCP), the following considerations should be taken into account:
Requirements:
- Size of Data: The database is 20 TB in size, which requires a scalable solution capable of handling large volumes of data.
- Transaction Data: Since the data is operational system transaction data, ACID compliance and transactional consistency are critical.
- Fully Managed: The database should be fully managed to simplify maintenance and reduce administrative overhead.
Option A: Cloud SQL
- Scalability: Cloud SQL supports vertical scaling (increasing instance size) and can scale up to a maximum of 64 TB of storage depending on the database engine (e.g., MySQL, PostgreSQL, SQL Server).
- Transactionally Consistent: Cloud SQL supports ACID transactions, making it a good choice for transaction data.
- Size: 20 TB can be handled within Cloud SQL, but it's not the most ideal choice for large-scale operational databases.
- SQL Support: Cloud SQL supports SQL and is fully managed, which is ideal for relational workloads.
- Why Rejected: While Cloud SQL can handle 20 TB and transactional consistency, it does not support automatic horizontal scaling, which may become a bottleneck as the database grows. It’s better suited for smaller to mid-sized workloads or less demanding scaling requirements.
Option B: Cloud Bigtable
- Scalability: Cloud Bigtable is designed for massive horizontal scalability and can handle petabytes of data.
- Transactionally Consistent: Cloud Bigtable is not ACID-compliant. It offers eventual consistency and is primarily used for high-throughput, low-latency workloads, not transactional data.
- Size: Cloud Bigtable can scale far beyond 20 TB, but it's primarily designed for NoSQL use cases like time-series data or real-time analytics, not relational transaction data.
- SQL Support: Cloud Bigtable does not support SQL; it uses its own query language for NoSQL data models.
- Why Rejected: Cloud Bigtab...
Author: Isabella · Last updated Jul 15, 2026
You need to choose a database to store time series CPU and memory usage for millions of computers. You need to store this data in one-second interval samples. Analysts will be performing real-time, ad hoc analytics against the database. You want to avoid being charged for every query executed ...
Let's analyze each option based on key factors like cost efficiency, scalability, schema flexibility, performance, and query requirements.
A) Create a table in BigQuery, and append the new samples for CPU and memory to the table
- Pros:
- BigQuery is designed for large-scale analytics and can efficiently handle time-series data. The append-only model is straightforward to implement.
- Good for real-time analysis when aggregated or analyzed over a larger time window.
- No need to update existing rows; each sample becomes a new row, which is cost-effective in terms of writes.
- Querying historical data is efficient if properly partitioned and indexed.
- Cons:
- The major downside is the query cost in BigQuery. BigQuery charges for the data processed during a query, which can be quite high if you're running frequent queries across millions of records.
- Managing the sheer volume of data could be cumbersome over time, especially with data stored at the second-level granularity.
- Use case: This approach is ideal for batch processing or when queries focus on aggregating large amounts of historical data.
B) Create a wide table in BigQuery, create a column for the sample value at each second, and update the row with the interval for each second
- Pros:
- This could reduce the data size per query, since only one row is updated per minute, which may reduce some cost associated with storing time-series data.
- Cons:
- BigQuery is not designed to efficiently handle frequent updates, especially for a wide table like this. The frequent updates could lead to performance bottlenecks and inefficiencies, and managing this kind of structure in BigQuery might introduce issues around data consistency and complexity in managing large datasets.
- In addition, BigQuery has a maximum row size, which could be problematic if you store multiple columns for each second within a single row.
- Use case: This approach might be used for small-scale scenarios where low-frequency queries or batch-style analysis are needed, but it’s not ideal for large-scale time-series data.
C) Create a narrow table in Bigtable with a row key that combines the Computer Engine computer identifier with the sample time at each second
- Pros:
- Bigtable is a great option for high-performance, low-latency operations on time-series data. The narrow table design makes it flexible and scalable.
- Bigtable is optimized for quick reads a...
Author: SilverBear · Last updated Jul 15, 2026
You want to archive data in Cloud Storage. Because some data is very sensitive, you want to use the `Trust No One` (TNO) approach to encrypt your data to prevent t...
To achieve the "Trust No One" (TNO) encryption approach, the goal is to ensure that the cloud provider itself cannot decrypt your data. This requires the encryption keys to be controlled entirely by you, and the provider should not have access to these keys. Let's analyze each option based on key factors such as security, control over encryption keys, and cloud provider access.
A) Use gcloud kms keys create to create a symmetric key. Then use gcloud kms encrypt to encrypt each archival file with the key and unique additional authenticated data (AAD). Use gsutil cp to upload each encrypted file to the Cloud Storage bucket, and keep the AAD outside of Google Cloud.
- Pros:
- This method allows you to use a symmetric key for encryption, which is managed by Google Cloud Key Management Service (KMS).
- Keeping the AAD outside of Google Cloud ensures that the key and data remain under your control, and Google cannot access the data as they won't have the AAD.
- Cons:
- While Google Cloud cannot decrypt the data, you need to manage the AAD securely, which requires additional management overhead.
- Google will still control the encryption key lifecycle in KMS, and unless you control the KMS keys (with hardware security modules, for example), there's still potential access to the key by Google.
- Use case: This is good for maintaining control over encryption but requires managing sensitive metadata (AAD) outside of the cloud.
B) Use gcloud kms keys create to create a symmetric key. Then use gcloud kms encrypt to encrypt each archival file with the key. Use gsutil cp to upload each encrypted file to the Cloud Storage bucket. Manually destroy the key previously used for encryption, and rotate the key once.
- Pros:
- Destroying the key after encryption would mean that no one, including you, can decrypt the files unless they have access to the key.
- Rotating the key provides an additional layer of security.
- Cons:
- Destroying the encryption key would make it impossible to decrypt the data in the future unless another copy of the key is stored somewhere.
- This could be impractical in the long term because you would lose the ability to access the data unless very careful key management is practiced.
- Use case: This is more useful for data that is truly archived and unlikely to need decryption, but it could cause problems if you ever need to retrieve the data.
C) Specify customer-supplied encryption key (CSEK) in the .boto configuration file. U...
Author: Ava · Last updated Jul 15, 2026
You have data pipelines running on BigQuery, Dataflow, and Dataproc. You need to perform health checks and monitor their behavior, and then notify the team managing the pipelines if they fail. You also need to be able to work across multipl...
To monitor your data pipelines across multiple projects and notify the team in case of failures, the goal is to use a solution that allows centralized health checks, is easy to manage, integrates with existing GCP services, and offers scalability. Let's go through each option.
A) Export the information to Cloud Monitoring, and set up an Alerting policy
- Pros:
- Cloud Monitoring (formerly Stackdriver) is a fully managed service designed for monitoring and alerting on GCP resources, and it can integrate with various Google Cloud products like BigQuery, Dataflow, and Dataproc.
- It is a native Google Cloud product, so it supports all GCP services out of the box with minimal configuration.
- Cloud Monitoring supports setting up alerting policies that notify users via multiple channels (email, SMS, Slack, etc.) when certain metrics or logs trigger a failure condition.
- It supports monitoring across multiple projects, and you can configure alerts across various GCP projects in a centralized way.
- This approach is highly scalable, low-maintenance, and managed, which is ideal for an environment with many data pipelines.
- Cons:
- It may require some upfront setup to filter and configure the right metrics and logs, but once set up, it is very effective.
- Use case: This is the best option for monitoring GCP services because it is fully managed, integrated with other GCP services, and scalable across projects with minimal effort.
B) Run a Virtual Machine in Compute Engine with Airflow, and export the information to Cloud Monitoring
- Pros:
- Airflow is useful for orchestrating workflows and could potentially help automate health checks.
- You could also export relevant information to Cloud Monitoring.
- Cons:
- Managing a Compute Engine VM and Airflow increases the operational overhead significantly. Airflow would need to be manually set up, maintained, and scaled.
- This introduces complexity, as you would need to manage the VM, Airflow tasks, and Cloud Monitoring integrations. Additionally, Airflow would be redundant if your main requirement is just health checks and notifications.
- It doesn't align with the preference for using managed products or services, as managing VM instances and Airflow requires more hands-on intervention and resource management.
- Use case: Suitable if you need a custom orchestration solution and are okay with the complexity of managing it, but it is not ideal for simple health checks and notifications.
C)...
Author: Emily · Last updated Jul 15, 2026
You are working on a linear regression model on BigQuery ML to predict a customer's likelihood of purchasing your company's products. Your model uses a city name variable as a key predictive component. In order to train and serve the model, your data must be organized in columns. You ...
In this scenario, you are working on a linear regression model using BigQuery ML to predict customer purchase likelihood, with city name as a key predictive component. You need to prepare the data for training while ensuring minimal coding effort and preserving predictive variables (city names). Let’s break down the options:
A) Create a new view with BigQuery that does not include a column with city information.
- Pros:
- Simple to implement using a view in BigQuery.
- The transformation of the data is straightforward.
- Cons:
- City information is a key predictive component of your model. Excluding it would defeat the purpose of using the city variable as a predictor.
- This option is not viable because it eliminates the predictive feature (city) from the model, reducing the effectiveness of your model.
- Use case: This option would be suitable if you need to exclude irrelevant or non-predictive features, but in this case, it contradicts the need to use city data as a predictor.
B) Use SQL in BigQuery to transform the state column using a one-hot encoding method, and make each city a column with binary values.
- Pros:
- One-hot encoding is a common technique to handle categorical variables like city names, transforming them into a format that can be used in regression models (creating binary columns for each city).
- This approach is highly effective for a variable like city, where each city represents a distinct category.
- It ensures that you retain the predictive power of the city variable while transforming it into a format that can be easily used in the model.
- Cons:
- If the dataset contains a large number of unique cities, one-hot encoding may lead to an extremely large number of columns, which can increase the size of the data and slow down model training. However, for manageable datasets with fewer cities, this would work well.
- Use case: This is a good choice for small-to-medium datasets where one-hot encoding does not result in excessive column proliferation. This option is easy to implement in BigQuery SQL with minimal coding and is widely used in regression models.
C) Use TensorFlow to create a categorical variable with a vocabulary list. Create the vocabulary file and upload that as part of your model to BigQuery ML.
- Pros:
- Using TensorFlow for encoding categorical variables is powerful and flexible.
- This approach could ...
Author: Zain · Last updated Jul 15, 2026
You work for a large bank that operates in locations throughout North America. You are setting up a data storage system that will handle bank account transactions. You require ACID com...
In this scenario, you're setting up a data storage system for handling bank account transactions. The key requirements are:
1. ACID compliance (Atomicity, Consistency, Isolation, Durability) — to ensure the integrity and reliability of transaction data.
2. Ability to access data with SQL — to easily query and manage transaction data.
Let's review the options:
A) Store transaction data in Cloud Spanner. Enable stale reads to reduce latency.
- Pros:
- Cloud Spanner is designed to provide ACID compliance and can scale horizontally across regions, which is useful for large banks operating in multiple locations.
- It supports SQL queries through its own SQL interface, so it meets the requirement for SQL access.
- Stale reads could help reduce latency in certain cases by reading data that may not be immediately consistent but is eventually consistent, which can improve performance in some scenarios.
- Cons:
- Stale reads are typically used to improve read performance by allowing a slight inconsistency in the data. This may not be appropriate for transaction data, as transactions often require strong consistency. Allowing stale reads could compromise the integrity of transaction records, which is unacceptable in banking.
- Use case: This is not ideal for handling transaction data that requires ACID guarantees because stale reads may result in inconsistent data. This would not meet the requirement of strict ACID compliance.
B) Store transaction in Cloud Spanner. Use locking read-write transactions.
- Pros:
- Cloud Spanner is fully ACID-compliant, and it supports SQL access.
- By using locking read-write transactions, you can ensure that transactions are properly isolated, and ACID properties are maintained.
- This approach ensures that transactions are consistent, and the locking mechanism prevents issues like double spending or conflicting transactions in a banking context.
- Cons:
- While Cloud Spanner is well-suited for ACID-compliant operations, locking read-write transactions may introduce performance overhead, especially if there are frequent updates. However, for banking systems where transaction integrity is critical, this trade-off is acceptable.
- Use case: This is the best option for ensuring ACID compliance and SQL access while handling sensitive bank account transaction data....
Author: Liam · Last updated Jul 15, 2026
A shipping company has live package-tracking data that is sent to an Apache Kafka stream in real time. This is then loaded into BigQuery. Analysts in your company want to query the tracking data in BigQuery to analyze geospatial trends in the lifecycle of a package. The table was originally created with ingest-date partitioning. Over ti...
Let's break down the different options and evaluate each in terms of query performance, use cases, and why they may or may not work well in this scenario.
A) Implement clustering in BigQuery on the ingest date column.
- Clustering by ingest date would not significantly improve query performance. This is because the queries most likely focus on tracking data over time (such as the package delivery date, location, or other aspects), and clustering by the ingest date would organize the data based on when it was ingested, not when the package is actually being tracked.
- Reason for rejection: The analysts are concerned with geospatial trends and tracking over time, so clustering by ingest date would not help. Queries that focus on the actual delivery date or geographical data would not benefit from clustering by ingest date.
B) Implement clustering in BigQuery on the package-tracking ID column.
- Clustering by package-tracking ID can help improve query performance when searching for specific packages by their tracking ID, but it may not help with the kind of geospatial analysis the analysts are interested in. For geospatial queries, you'd likely need to filter or group data by location or delivery date, not just the tracking ID.
- Reason for rejection: This option would optimize querying individual packages but not improve performance on queries that require analysis of trends over time or geospatial patterns.
C) Tier older data onto Cloud Storage files and create a BigQuery table using Cloud Storage as an external data source.
- Using external tables in Cloud Storage can help reduce BigQuery costs for older d...
Author: David · Last updated Jul 15, 2026
Your company currently runs a large on-premises cluster using Spark, Hive, and HDFS in a colocation facility. The cluster is designed to accommodate peak usage on the system; however, many jobs are batch in nature, and usage of the cluster fluctuates quite dramatically. Your company is eager to move to the cloud to reduce the overhead associated with on-premises infrastructure and maintenance and to benefit from the cost savings. They are also hoping to modernize their existing infrastructure to use more serverless offerings in order to take advantage of the cloud. Because of the timing of their contrac...
Let's analyze each of the options in terms of migration strategy, cost-effectiveness, speed of execution, and alignment with cloud-native serverless offerings.
A) Migrate the workloads to Dataproc plus HDFS; modernize later.
- Dataproc is a fully managed Spark and Hadoop service in the cloud that can quickly accommodate the on-premises workloads. However, migrating to HDFS on Cloud Storage doesn't take full advantage of cloud-native features like scalability and cost efficiency.
- This approach would delay the modernization aspect, but it offers a quick migration path with minimal changes to the current infrastructure. It would be suitable if the company needs a quick lift-and-shift migration with limited modification in the short term.
- Reason for rejection: Although this provides a quick solution, sticking with HDFS would not fully capitalize on the cloud’s serverless capabilities, and the company is specifically looking to reduce overhead and embrace cost savings, which HDFS does not optimize.
B) Migrate the workloads to Dataproc plus Cloud Storage; modernize later.
- Cloud Storage is a more cloud-native, scalable, and cost-effective option compared to HDFS. By using Dataproc with Cloud Storage, the company can modernize the data storage layer by utilizing a scalable object storage system that reduces storage costs compared to traditional HDFS.
- This approach would allow a relatively quick migration with the potential for modernization later (e.g., migrating to serverless services). The use of Cloud Storage would allow the company to start optimizing their storage and reduce infrastructure maintenance overhead.
- Reason for selection: This option offers a faster, more cost-effective migration path that can make use of cloud-native tools. The company can modernize further after the initial migration, and it addresses both their need for speed and for reducing on-premises overhead.
C) Migrate the Spark workload to Dataproc plus HDFS, and modernize the Hive workload for BigQuery.
- This approach combines the quick migration of Spark workloads to Dataproc with BigQuery for Hive workloads. BigQuery is a fully managed serverless data warehouse, making it an attractive option ...
