Troubleshooting¶
Start with the request's conditions and observed state. A blocked replica is not necessarily a crashed operator; it often means a required permission, dependency, or safety condition is missing.
kubectl -n replica-lab get clusterreplica demo -o yaml
kubectl -n replica-lab get replicaaccesses
kubectl -n replicove-system logs deployment/replicove --tail=150
kubectl -n replica-lab get pods,pvc,services
Check the host context before running commands. Avoid dumping Secrets or kubeconfigs into logs or public issues.
Installation problems¶
| Symptom | Check / next action |
|---|---|
Operator ImagePullBackOff |
Check node access to ghcr.io and the selected alpha tag/digest. Published images are public for amd64/arm64; custom builds need a reachable registry or a loaded local test image. |
| Bootstrap Job fails | Read kubectl -n replicove-system logs job/replicove-bootstrap. Confirm Secret get/list/create permissions. An invalid existing key must be restored deliberately, not overwritten. |
| State exists but key is missing | Restore the original key and encrypted records from the same backup. Re-running bootstrap cannot recover lost encryption material. |
| Job update reports an immutable template | After confirming it is completed, remove only replicove-bootstrap and reapply the reviewed installer. Preserve the key. |
| “No matches for kind ClusterReplica” | Apply config/crd/ and wait for CRDs to be established before requests. |
| Persistent vCluster PVC is Pending | Check the default StorageClass, provisioner, quota, scheduling and capacity. The profile requests 1 GiB for control-plane state. |
| cert-manager is rolled out but source admission says connection refused | Deployment readiness can precede webhook Service routing or CA propagation. Wait for a server-side dry run of the fixture to succeed before creating Issuers/Certificates. The workload test bounds this wait and still fails invalid manifests or a persistently unavailable webhook. |
Planning and replication problems¶
If you are adding data copies to a running installation, use enable mirroring later. Common opt-in problems are:
| Symptom | Check / next action |
|---|---|
Mirror kind missing, or ReplicaGrant.spec.mirror is unknown |
Apply all matching release CRDs before upgrading the operator; Helm does not upgrade them. |
| New chart still runs an older operator | A saved image.digest takes precedence over the tag. Explicitly select the matching release image in the upgrade overlay. |
| Mirrors enabled but selected workload reads fail | mirrors.sources supplies snapshot permissions only. Preserve/add normal sources read rules and the exact administrator grant. |
| Snapshot APIs exist but snapshots never become ready | Check the existing controller and CSI snapshot/restore support. auto reuses existing infrastructure; it cannot repair an incomplete host installation. |
RuntimeNamespaceInUse |
The pinned vCluster permits one runtime per host namespace. Use a separate granted destination or an explicitly registered suitable existing target; enabling mirrors does not remove ordinary replicas. |
| A broad refresh tries to capture Replicove mirror RBAC | Upgrade to 0.2.0-alpha.2, which marks all bundled mirror infrastructure as excluded from application capture. |
| Symptom / reason | Check / next action |
|---|---|
GrantChanged |
Delete and recreate the request after cleanup under the revised administrator grant. |
| Source reads denied | Check both host source RBAC and grant kinds/namespaces. A grant cannot create API permissions. |
| Missing or excluded dependency | Include/grant its known dependency, or adjust the source application and create a new request. |
| Secret not selected | Check the exact Secret grant and Snapshot/Follow; token and bootstrap Secret types remain excluded. |
| Capture/object limit exceeded | Narrow the selection or split tests. Increase grant limits only within API bounds and measured capacity. |
| Chart lifecycle hook blocks capture | Disable only a documented optional hook with the correct chart value. Unsupported required hooks need an adapter. |
AwaitingApproval |
Inspect status.plan, then approve exactly status.plan.revision. Refresh requires a new approval. |
PlatformQualificationRequired |
Platform provisioning is not qualified. Use an administrator-approved owned Helm runtime or registered existing target where policy permits. |
ProfileMismatch / target identity conflict |
Restore the intended runtime/credential and inspect the pinned UID. Do not redirect the request to another cluster under the same name. |
| Custom resource never Ready | Check operator Deployment/CRD readiness and the configured guest condition/status-field check. |
Access problems¶
A session needs a Ready replica, the correct current replica UID, a granted role, and sufficient remaining lifetime. Create it with 600–3,600 seconds, within the administrator limit. After a replica is recreated, old UIDs must not be reused.
Forbidden while retrieving the session Secret means the host caller lacks exact-name read permission. Check accessSubjects or the administrator-created binding. Do not solve it by granting access to every Secret in the destination namespace.
A local TLS error often means the endpoint was changed without retaining the guest's CA or TLS server name. The CLI manages these fields. For manual port-forwarding, update only the server address in the issued kubeconfig. A refused localhost connection usually means the forward stopped or the selected port is occupied.
For existing targets, the operator and consumer both need appropriate routes. CLI connect is for owned runtimes; use access with an existing target's configured endpoint.
Deletion appears stuck¶
Inspect the request's condition reason, guest API availability, protected state, workload finalizers, and relevant PVC/PV status. Keep the operator running. Retain reclaim behavior, an unreachable target, missing state, or a reused foreign UID can prevent verification. See cleanup recovery.
A PVC waits for a guest test Pod¶
Check for guest Pods that still reference the copied claim, including completed probe Pods left by integration tests. Published v0.3.0-alpha.1 has a cleanup-ordering defect in this case: it can wait for the PVC before deleting the replica-owned namespace containing its extra consumer. Upgrade to v0.3.0-alpha.2, whose fix passed exact-source and published-artifact checks; see the versioned validation record.
In an exclusively owned guest namespace, alpha.2 requests namespace deletion while that ordinary PVC remains pending, then wait for normal Kubernetes cleanup and the recorded volume checks. Deprovisioning includes guest-created workloads inside the disposable namespace. It still preserves ownership checks, finalizers and controller/custom-resource ordering. See guest-created volume consumers.
In any pre-existing guest namespace reused without adoption, the namespace and foreign Pods are preserved during guest-resource cleanup. This includes default in a managed guest as well as borrowed existing-target namespaces. Its administrator must remove or reconfigure the exact foreign consumer before the owned PVC can finish deleting. Do not delete the shared namespace as a workaround. On alpha.1, an administrator can similarly remove the intended disposable test Pod through an already authorized guest route and let cleanup retry. New Replicove access sessions are unavailable once cleanup starts; use the administrator's existing route rather than expanding requester permissions. Never remove PVC protection, protected state or Replicove finalizers to hide this wait.
Report a reproducible problem¶
Include the commit, host Kubernetes version, profile/guest version, installation method, sanitized request/grant YAML, and condition reason. For a CI failure, link the failing job. Remove credentials, source Secret values, connection material, and private endpoints before sharing. Use Support for bugs/questions and Security for vulnerabilities.