How to Connect a Public Cloud Run V2 Service to an Internal-Only Service Using Terraform?

I have two Cloud Run V2 services deployed via Terraform: a publicly accessible Server App and an Embedding Generator App that should only be accessed internally by the Server App. I've set up a subnet and VPC connector to facilitate this internal connectivity.

The problem: when the Server App calls the Embedding Generator App, I receive a 404 error and no logs appear on the Embedding Generator side. However, when I test the connection from a VM attached to the Horcrux subnet, the call to the Embedding Generator succeeds. This leads me to suspect an issue in how the network connectivity or service endpoints are configured.

Below are my Terraform files for the services and the VPC connector:

embedding_generator_app.tf

HCLCopied
resource "google_service_account" "embedding_generator_app" {
  account_id = "embedding-generator-app"
}

resource "google_cloud_run_v2_service" "embedding_generator_app" {
  name     = "embedding-generator-app"
  location = "us-central1"
  ingress  = "INGRESS_TRAFFIC_INTERNAL_ONLY"

  template {
    service_account = google_service_account.embedding_generator_app.email

    scaling {
      max_instance_count = 10
    }

    vpc_access {
      connector = google_vpc_access_connector.horcrux.id
      egress    = "PRIVATE_RANGES_ONLY"
    }

    containers {
      image = "project-registry/embedding-generator-app"

      ports {
        container_port = 1010
      }

      resources {
        startup_cpu_boost = true
        limits = {
          cpu    = "4000m"
          memory = "2Gi"
        }
      }
    }
  }
}

resource "google_cloud_run_v2_service_iam_binding" "embedding_generator_app_run_invoker" {
  name     = google_cloud_run_v2_service.embedding_generator_app.name
  project  = google_cloud_run_v2_service.embedding_generator_app.project
  location = google_cloud_run_v2_service.embedding_generator_app.location
  role     = "roles/run.invoker"
  members = [
    "allUsers",
  ]
}

server_app.tf

HCLCopied
resource "google_service_account" "server_app" {
  account_id   = "server-app"
}

resource "google_service_account_key" "server_app" {
  service_account_id = google_service_account.server_app.name
}

resource "google_cloud_run_v2_service" "server_app" {
  name     = "server-app"
  location = "us-central1"
  ingress  = "INGRESS_TRAFFIC_ALL"

  template {
    service_account = google_service_account.server_app.email

    scaling {
      max_instance_count = 10
    }

    vpc_access {
      connector = google_vpc_access_connector.horcrux.id
      egress    = "PRIVATE_RANGES_ONLY"
    }

    containers {
      image = "project-registry/serverapp"

      ports {
        container_port = 9090
      }

      resources {
        startup_cpu_boost = true
        cpu_idle          = true
        limits = {
          cpu    = "2000m"
          memory = "1Gi"
        }
      }

      env {
        name  = "EMBEDDING_GENERATOR_APP_URL"
        value = google_cloud_run_v2_service.embedding_generator_app.uri
      }
    }
  }
}

resource "google_cloud_run_v2_service_iam_binding" "server_app_run_invoker" {
  name     = google_cloud_run_v2_service.server_app.name
  project  = google_cloud_run_v2_service.server_app.project
  location = google_cloud_run_v2_service.server_app.location
  role     = "roles/run.invoker"
  members = [
    "allUsers",
  ]
}

vpc.tf

HCLCopied
resource "google_project_service" "vpc_access_api" {
  project    = "project-id"
  service = "vpcaccess.googleapis.com"
}

resource "google_compute_subnetwork" "horcrux" {
  name          = "horcrux"
  ip_cidr_range = "10.2.0.0/28"
  region        = "us-central1"
  network       = "default"
  private_ip_google_access = true
  depends_on = [google_project_service.vpc_access_api]
}

resource "google_vpc_access_connector" "horcrux" {
  name          = "horcrux"
  machine_type  = "e2-micro"
  min_instances = 2
  max_instances = 3
  subnet {
    name = google_compute_subnetwork.horcrux.name
  }
}

I'm seeking insights or suggestions on any potential issues in my Terraform configuration or network setup that might be causing this connectivity error.