Diagnose & validate your Infrastructure

Guide topics:

Diagnose your infrastructure

This page provides guidance on how to troubleshoot and validate your Aggregation Service infrastructure. Follow the steps to identify and resolve issues.

Troubleshooting Terraform

Terraform, the infrastructure as code (IaC) platform created by HashiCorp, is the main driver for deploying Aggregation Service to your chosen public cloud. If you don't have terraform installed, check out HashiCorp documentation on how to obtain the cli locally.

You may also consider installing tfenv, if your needs require you to switch terraform versions.

If you do not see a resolution to your Aggregation Service issue, notify us by filing a GitHub issue or submitting the technical support form.

If you would rather switch terraform manually:

rm -rf $(which terraform)
wget https://releases.hashicorp.com/terraform/1.2.3/terraform_1.2.3_linux_amd64.zip 
unzip terraform_1.2.3_linux_amd64.zip
mv terraform /usr/local/bin/terraform
terraform --version

If a Terraform error is unclear, the output can be viewed by enabling Terraform logging at TRACE level.

Google Cloud

The Terraform provider for Google Cloud is developed jointly between HashiCorp and Google. For further questions and troubleshooting advice, reach out to Cloud Terraform support for more information or head on over to the Google section of the Hashicorp Discuss portal.

Amazon Web Services

Debugging documentation can be found on AWS site. We also encourage you to check out the AWS section of the HashiCorp Discuss portal.

For further questions about Terraform, see the HashiCorp community portal. Troubleshooting advice for general Terraform, read up on their debugging documentation.

Take great care when you modify these directories, because the removal of them will cause terraform to not recognize the state any more. It will attempt to re-provision all the resources and error out when you have a resource of the exact same name. If this continues to be an issue, reach out to us in the technical support form.

Take great care when you modify these directories, because the removal of them will cause terraform to not recognize the state any more. It will attempt to re-provision all the resources and error out when you have a resource of the exact same name. If this continues to be an issue, reach out to us in the technical support form

Validate your Cloud Setup

Verify if your Aggregation Service Infrastructure in your chosen public cloud has been deployed properly. To do this, we will use sample debug reports and run a job in debug mode. If Aggregation Service is able to receive the API request and respond, we know that it's up and running. If you have not set one up already, we recommend you utilize Postman or cURL to set up your API calls to Aggregation Service. Note that debug modes should be switched off, set to false, during your normal aggregation operations.

In addition to the following steps, to further validate your Aggregation Service infrastructure, investigate all of the cloud components that get provisioned in your public cloud.

Steps

  1. To generate sample debug reports and output domain files, follow the relevant steps of the codelabs:

  2. Run a createJob request with this body, replacing the highlighted placeholders:

    {
  "job_request_id": "<job_request_id>",
  "input_data_blob_prefix": "<report_folder>/<report_name>.avro",
  "input_data_bucket_name": "<input_bucket_name>",
  "output_data_blob_prefix": "<output_folder>/<summary_report_prefix>",
  "output_data_bucket_name": "<output_bucket_name>",
  "job_parameters": {
    "output_domain_blob_prefix": "<output_domain_folder>/<output_domain>.avro",
    "output_domain_bucket_name": "<output_domain_bucket_name>",
    "attribution_report_to": "<reporting_origin_domain>",
    "report_error_threshold_percentage": "10", // example value
    "debug_run": "true" // optional, but should be false by default
  }
}

  3. Run a getJob request to see the response. If you're getting the following response, it means your Aggregation Service was deployed properly.

    {
    "job_status": "FINISHED",
    "request_received_at": <utc_timestamp>,
    "request_updated_at": <utc_timestamp>,
    "job_request_id": "<job_request_id>",
    "input_data_blob_prefix": "<report_folder>/<input_report_prefix>.avro",
    "input_data_bucket_name": "<input_bucket_name>",
    "output_data_blob_prefix": "<output_folder>/<summary_report_prefix>",
    "output_data_bucket_name": "<output_bucket_name>",
    "postback_url": "",
    "result_info": {
        "return_code": "<return_code_and_message>",
        "return_message": "Aggregation job successfully processed",
        "error_summary": {
            "error_counts": [],
            "error_messages": []
        },
        "finished_at": <utc_timestamp>
    },
    "job_parameters": {
        "debug_run": "true", // optional, but should be false by default
        "output_domain_bucket_name": "<output_domain_bucket_name>",
        "output_domain_blob_prefix": "<output_domain_folder>/<output_domain>.avro",
        "attribution_report_to": "<reporting_origin_domain>"
    },
    "request_processing_started_at": <utc_timestamp>
}

If you do not receive a response with a job_status: FINISHED, collect more information by navigating to the compute engine or EC2 resource of your chosen public cloud. Inspect your Aggregation Service instance and inspect the logs and record the information that you see between TEE container launcher initiating and TEE container launcher exiting to provide to our technical support team.

Google Cloud

  1. Navigate to Google Cloud: Compute Engine.
  2. Click VM instances and select your Aggregation Service instance.
  3. Click Details > Logs > Logging (or search for Logs Explorer).

Amazon Web Services

  1. Navigate to Amazon Web Services: EC2 Enclave.
  2. Click EC2 instances and select your Aggregation Service instance.
  3. Click Actions > Monitor and troubleshoot > Get system log.

Inspect and see if you find any matching the error notifications in our public guidance on GitHub.

Note: If you have just upgraded your current version of Aggregation Service, the full provisioning process may take up to a couple of hours for the correct instance image to reflect in your final compute service instance. Reach out to us if this isn't the case for you.

Next Steps

If you do not see a resolution to your Aggregation Service issue, notify us by filing a GitHub issue or submitting the technical support form.