🔧
Network Troubleshooting — Why It Matters
Acting as an SME advocate — building evidence to get the network team to investigate

The Real-World Challenge

As a Commvault SME or backup admin, you will encounter situations where backup jobs fail or a media agent goes offline intermittently. Log analysis points to a network connectivity issue — but when you escalate to the network team, they say "everything is fine on our end" and refuse to investigate further.

🎯 Your Role as Backup SME

You cannot simply show Commvault log error messages to the network team — they need solid, tool-generated evidence. Your job is to run the right diagnostic tools, capture the output, and present it as proof that forces the network team to begin their own investigation. The tools below give you exactly that evidence.

Common Connectivity Issues in Commvault Environments

📡 Commvault Entity Connectivity Map
All network paths where connectivity issues can occur between Commvault components
🖥 CommServe Management Server
Port 8400 / 8403
↙     ↓     ↘
💾 Media Agent Data Mover
Storage Target
🔄 VSA Proxy Access Node
Snapshot Handler
📂 Client FS / SQL Agent
Application Data
↕         ↕
☁ Cloud Storage Azure / AWS / GCP
Cloud Library Target
🏗 ESXi / Hypervisor VMware / Hyper-V
VM Snapshot Source

Connectivity problems can occur between any two Commvault entities. These include:

CommServe ↔ Media Agent CommServe ↔ VSA Proxy CommServe ↔ Client Media Agent ↔ Client Media Agent ↔ VSA Proxy VSA Proxy ↔ ESXi Host

Common Root Causes of Network Issues

  • Firewall blocking ports: Windows Defender Firewall or network firewall blocking Commvault ports 8400/8403
  • NSG (Network Security Group) rules: For cloud-hosted agents, Azure/GCP/AWS NSG rules may block traffic
  • MTU mismatch: One server using MTU 9000 (jumbo frames), another using MTU 1500 — causes TCP resets
  • Degraded hardware: Failing NIC card, loose cable, faulty switch port, or degraded NIC firmware/drivers
  • Intermittent TCP resets: Connection between two entities is being reset, causing media agent to go offline intermittently or backup jobs to go into pending state
  • Wrong IP address: Human error — incorrect destination IP configured (seen in lab demo)

Toolkit Overview — All Network Diagnostic Tools

Tool Purpose Platform Use Case
Ping Basic reachability check — packet loss, latency Windows & Linux First check — is the host even reachable? May be disabled (ICMP blocked)
Telnet Port-level connectivity test Windows & Linux Verify ports 8400/8403 are open and reachable. May be blocked by security policy
CVPing Commvault-native port connectivity test — wrapper of telnet Windows & Linux When telnet is blocked, use CVPing. Bundled with every Commvault agent install
Netstat Verify local port is listening Windows & Linux Confirm port 8400 is actually in LISTENING state on the target machine
CV Network Test Tool Measure network throughput (bandwidth) between two Commvault entities Windows (GUI) & Linux (CLI) Backup slowness investigation — prove bandwidth is insufficient to network team
Traceroute / Tracert Show hop-by-hop routing path between two hosts Linux (traceroute) & Windows (tracert) Show network team how many hops between servers and where latency is introduced
Wireshark Packet-level capture and analysis — find TCP resets and connection drops Windows Last resort when all other tools show "OK" but intermittent failures persist
TCPDump Linux equivalent of Wireshark — command-line packet capture Linux Capture packets on Linux Media Agents, analyze offline using Wireshark
📶 Network Diagnostic Escalation Ladder
Bottom-up approach — start simple, escalate only when needed
L1
Ping — Basic ICMP Reachability Is the host alive? Check latency & packet loss. May be blocked (ICMP disabled).
L2
Netstat — Verify Port Listening Locally Is port 8400 in LISTENING state on the target? If not, services aren't running.
L3
CVPing / Telnet — Port-Level TCP Connectivity Can you reach port 8400/8403 from the source machine? CVPing ships with every Commvault install.
L4
Traceroute — Hop-by-Hop Path Analysis How many hops? Where is latency introduced? Show to network team.
L5
CV Network Test Tool — Throughput Measurement What is the actual bandwidth (MB/s)? Run both directions. Uses port 25000.
L6
Wireshark / TCPDump — Deep Packet Inspection Last resort. Find TCP RST (connection resets). Requires downtime window. Share .pcap with network team.
📡
Ping, Telnet, Netstat & CVPing
First-line connectivity checks — verifying ports and host reachability

Ping — Basic Reachability

The most common starting point. Use ping to verify basic IP connectivity between two Commvault entities. It also reveals packet loss and intermittent timeouts, which indicate dropped packets on the network path.

⚠ ICMP Often Disabled at Customers

Ping relies on the ICMP protocol, which most enterprise customers disable on their firewalls and network devices for security reasons. If ping is blocked, do not assume connectivity is broken — move to CVPing or telnet which use TCP ports instead.

cmd/bash — Ping Examples
# Basic ping from CommServe to Media Agent ping 192.168.1.110 # Continuous ping (useful for intermittent issues — captures drops) ping -t 192.168.1.110 # Windows: -t for continuous ping -c 100 192.168.1.110 # Linux: -c for count # What to look for in output: # "Request timed out" = packet dropped or ICMP blocked # "TTL expired" = routing loop or max hops exceeded # High latency (>100ms) on same LAN = network congestion or bad hardware

Netstat — Verify Port is Listening Locally

Before diagnosing external connectivity, always verify the Commvault service is actually listening on port 8400/8403 on the target machine. A port that isn't listening cannot be reached regardless of firewall settings.

cmd/bash — Netstat Commands
# Windows — check if port 8400 is listening netstat -enob | findstr 8400 # Switches: -e (ethernet stats), -n (numeric), -o (PID), -b (process name) # Look for: 0.0.0.0:8400 ... LISTENING # Linux — equivalent command netstat -tulpn | grep 8400 # Switches: -t (TCP), -u (UDP), -l (listening), -p (PID), -n (numeric) # Look for: 0.0.0.0:8400 ... LISTEN # If port is NOT listening: # Check if Commvault services are running # Restart: commvault restart (Linux) or services.msc (Windows)

CVPing — Commvault's Built-in Port Connectivity Tool

CVPing is a Commvault-provided binary that tests port-level TCP connectivity between two Commvault entities. It is the equivalent of telnet for port testing but is always available since it ships with every Commvault agent installation. Use it when telnet is blocked by security policy.

Binary Location Commvault Base Folder — the installation directory (e.g., C:\Program Files\Commvault\ContentStore\Base\ on Windows, /opt/commvault/Base/ on Linux) Base Folder
Default Port Tested Port 8400 by default if no port is specified. You can specify other ports explicitly. Port 8400 Default
Available On Every Commvault entity — CommServe, Media Agent, Client, VSA Proxy. Wherever a Commvault agent is installed. All Entities
cmd/bash — CVPing Usage
# Run CVPing from CommServe to Media Agent on port 8400 (default) cvping 192.168.1.110 # Run CVPing on a specific port cvping 192.168.1.110 -port 8403 # On Linux: run from base folder ./cvping 192.168.1.108 # Expected output when port is OPEN: # "Connected to 192.168.1.110 on port 8400 successfully" # Expected output when port is BLOCKED: # "Connection timed out" or "Connection refused" # Troubleshooting CVPing failure when services are running: # 1. Verify correct IP (human error is common — wrong IP gives same timeout) # 2. Check Windows Defender Firewall (may block even if NSG is open) # 3. Check network/corporate firewall between the two servers # 4. For cloud: check NSG/VPC firewall rules — ports 8400, 8403 bidirectional
⚠ Real-World Mistake — Wrong IP Address

In the lab demo, CVPing failed even though services were running and Defender was off. Root cause: the destination IP was entered as 192.168.1.160 instead of the correct 192.168.1.110. Human error like this is extremely common. Always double-check the IP using ipconfig (Windows) or ip addr (Linux) on the destination machine before escalating.

📘 Check Readiness in CommCell Console

Another quick option: right-click any client in the CommCell Console → Check Readiness. This internally performs a port connectivity check (8400/8403) from the CommServe to that client and displays a Pass/Fail result. It is a fast first-check before running manual CVPing.

💡 Firewall Escalation Path

If ports are blocked: (1) For Windows OS firewall — ask the OS/Windows team to open ports 8400 and 8403 as inbound rules in Windows Defender Firewall (or disable it temporarily with their consent). (2) For corporate/network firewall — raise a request with the network team to open those ports bidirectionally. (3) For cloud NSG (Azure/GCP/AWS) — ask the cloud team to add inbound rules for 8400 and 8403. Never take unilateral action on production firewalls without change management approval.

📊
CV Network Test Tool — Measuring Throughput
Identifying slow backups caused by insufficient network bandwidth between Commvault entities

What is the CV Network Test Tool?

The CV Network Test Tool measures the actual network throughput (bandwidth in MB/s) between two Commvault entities. It is used when backups are running slowly and you suspect the network is the bottleneck. The results give you concrete data to show the network team.

Windows GUI-based application. Located in the Commvault Base folder. Search for cvnetworktesttool and run as Administrator. GUI Available
Linux Command-line only (no GUI). Run as ./cvnetworktesttool from the Base folder. Case-sensitive filename. CLI Only
Default Port Used Port 25000. This port must be open between the two machines. In restricted environments where 25000 is blocked, stop Commvault services and temporarily use port 8400 or 8403 instead. Port 25000 Required
Good Throughput 156 MB/s or higher on a standard 1 GbE LAN (as demonstrated in the lab). Below 50 MB/s is considered slow; below 20 MB/s is severely degraded. 156 MB/s = Good
📘 How It Works

One machine runs in Server Mode (listens on port 25000), the other runs in Client Mode (connects to the server IP). The tool sends data back and forth and calculates throughput in MB/s. Always run in both directions (A→B and B→A) to get a complete picture — throughput can differ by direction due to asymmetric routing or NIC duplex mismatch.

📊 CV Network Test Tool — Client/Server Architecture
Bidirectional throughput testing between two Commvault entities
🖥 Machine A — Server Mode Bind IP: 192.168.1.110
Listening on Port 25000
e.g., Media Agent
← Data stream →
⇄
Port 25000
(or 8400 fallback)
🖥 Machine B — Client Mode Connects to Server IP
Initiates throughput test
e.g., CommServe
↕ swap roles
🖥 Machine A — Client Mode Connects to B
Reverse direction test
← Data stream →
⇄
Compare results
🖥 Machine B — Server Mode Listening on Port 25000
Reverse direction
📈 Result: 156 MB/s both directions = Healthy < 50 MB/s = Slow  |  < 20 MB/s = Severely Degraded  |  Asymmetric = Duplex mismatch

Running the Tool — Windows GUI

Windows Media Agent ↔ CommServe Throughput Test
1
Open Tool on Server Machine (e.g., Windows Media Agent)

Base Folder → search for cvnetworktesttool → Right-click → Run as Administrator. In the tool, go to the Network Check tab (skip Lookup and Service Check tabs). Select Server Mode. Enter the Bind IP — this is the IP address of the current machine on the backup network interface. Click Start. The server is now listening on port 25000.

2
Open Tool on Client Machine (e.g., CommServe)

Base Folder → open cvnetworktesttool as Administrator. Network Check tab → Select Client Mode. Enter the Server IP (IP of the machine running in Server Mode — the Media Agent). Leave port as 25000 default. Click Run Test.

3
Review Throughput Results

The tool displays the calculated throughput in MB/s. Run multiple iterations — do not rely on a single result. First iterations can be inaccurate (as seen in lab: first run showed 39 MB/s, second run showed the correct 156 MB/s). Take the consistent result across 2–3 runs as the actual throughput value.

4
Reverse Direction Test

Swap roles: CommServe in Server Mode, Media Agent in Client Mode. Run the test again. Compare results in both directions — they should be similar. If one direction is significantly slower, it points to a duplex mismatch or asymmetric routing issue on that path.

⭐ Always Test in Both Directions

Network issues can be directional. Always test A→B and B→A separately. In the lab, throughput was consistent at 156 MB/s in both directions — but in production with NIC duplex mismatches or asymmetric QoS policies, you will see different values per direction. Both results should be shared with the network team.

Running the Tool — Linux CLI

bash — Linux Media Agent: cvnetworktesttool CLI
# Navigate to the Commvault base folder cd /opt/commvault/Base/ # Run the tool (case-sensitive on Linux) ./cvnetworktesttool # Menu options presented: # 1 - Lookup (DNS resolution check) # 2 - Service Check # 3 - Network Check (Client Mode) — use this for throughput test # 4 - Network Check (Server Mode) — use this to listen # 5 - Exit # TO RUN AS SERVER (Linux Media Agent listens): # Select option 4 # Enter Bind IP: IP of this Linux machine (e.g., 192.168.1.108) # Tool listens on port 25000 — "Server is listening..." # TO RUN AS CLIENT (CommServe connects to Linux MA): # Select option 3 # Select option 1 to Execute # Enter Server IP: IP of the CommServe (e.g., 192.168.1.103) # Tool calculates throughput and displays result in MB/s # Run multiple iterations — first result may be inaccurate # Select option 3 again → option 1 to re-execute for next iteration
⚠ Port 25000 Workaround for Restricted Environments

Some customers only allow Commvault ports 8400 and 8403. Port 25000 will be blocked. Workaround: schedule a maintenance window, stop all Commvault services on both machines, then run the CV Network Test Tool using port 8400 (enter it manually instead of the default 25000). After the test, restart services on both machines. This is disruptive but necessary when 25000 is not permitted.

🦈
Wireshark & TCPDump — Deep Packet Inspection
Last-resort tool when all other checks pass but intermittent failures persist

When to Use Wireshark / TCPDump

Use Wireshark (Windows) or TCPDump (Linux) only after exhausting all other tools. The scenario: ping works, CVPing works, CV Network Test Tool shows good throughput, traceroute looks clean — but backups still fail intermittently, or the media agent goes offline randomly. This is when you need packet-level capture to find TCP connection resets.

🔍 What to Look for in Wireshark Output

TCP Retransmissions: These are normal and expected. The network team will tell you to ignore these — they do not indicate a problem by themselves. TCP Connection Resets (RST flag): These are critical and must be investigated. Frequent TCP resets between two backup entities will cause jobs to go into pending state, media agents to go offline, or intermittent disconnections. This is the evidence you bring to the network team.

Common Causes of TCP Resets

🦈 Wireshark Packet Analysis — What to Look For
Decision tree for interpreting captured traffic between Commvault entities
📦 Captured Packets (PCAP) Filter: ip.addr == <destination IP>
Inspect colour-coded rows
↙                   ↘
✅ TCP Retransmissions Normal & expected
Network team will say "ignore these"
NOT a problem indicator
🚨 TCP RST (Connection Reset) Critical — must investigate!
Dark/black rows in Wireshark
Root cause of intermittent failures
↓
Common Causes of TCP RST MTU mismatch (9000 vs 1500) · Degraded NIC hardware
Faulty cable / switch port · Outdated NIC drivers · QoS / ACL rules
  • MTU mismatch: One server using jumbo frames (MTU 9000) while another uses standard (MTU 1500). The mismatch causes packets to be dropped or reset
  • Degraded NIC hardware: Failing or degraded NIC card on a physical server — NIC firmware issues, hardware degradation
  • Faulty cable or switch port: Physical layer issues causing intermittent frame errors
  • Windows NIC driver issues: Outdated or corrupted NIC drivers causing reset behavior
  • Router or switch configuration: QoS policies, ACLs, or routing table issues
⚠ Real-World Warning — Wireshark Can Crash Degraded Servers

In a real customer incident, the instructor ran Wireshark on a CommServe that was already running slowly (degraded hardware). Starting Wireshark further loaded the server, causing it to hang and require a reboot — creating an unexpected outage. Always: (1) schedule a downtime window before running Wireshark, (2) involve the Windows/OS team — let them run the tool, (3) stop Commvault services before starting capture if possible, (4) never run Wireshark alone on a production server without on-call support available.

Wireshark — Windows Procedure

Running Wireshark to Capture Traffic Between CommServe and Media Agent
1
Install Wireshark on Both Machines

Download Wireshark installer from wireshark.org. Install on both the CommServe and the Media Agent (or whichever two entities you are investigating). Run the installer wizard with default options — it also installs npcap drivers. A reboot may be requested — schedule accordingly. If machines have no internet, download on a jump server and copy the executable.

2
Select the Correct Network Interface

Right-click Wireshark → Run as Administrator. On the main screen, click the black dot icon to list network interfaces. Identify the interface where your backup IP is configured (e.g., Ethernet 0). Do NOT capture all interfaces — limit to the backup NIC only. To confirm the NIC name: Control Panel → Network Adapters → note the name matching your backup IP.

3
Start Capture and Generate Traffic

Click the interface to start packet capture. Simultaneously, generate traffic between the two machines — run a backup job, or trigger a package install from CommServe to Media Agent. This creates packets to capture. Start Wireshark on both machines at the same time so you have captures from both ends of the connection.

4
Apply IP Filter and Analyze

In the Wireshark filter bar, enter: ip.addr == 192.168.1.110 (replace with the destination machine's IP). This filters to show only traffic between the two machines. Scroll through the results — dark/black highlighted rows indicate issues. Look specifically for "TCP RST" (connection reset) entries. Retransmissions are acceptable; resets are not.

5
Save as PCAP and Share

File → Save As → give a meaningful name (e.g., cs_to_ma110_backup_capture.pcap). Share the PCAP file with the network team or Commvault Technical Support if they request it. Commvault Support may ask for this file for offline analysis — they will not connect remotely for this step.

TCPDump — Linux Procedure

bash — TCPDump on Linux Media Agent
# Verify tcpdump is installed (available on most Linux distros by default) which tcpdump # If not found, install: yum install tcpdump (RHEL) or apt install tcpdump (Ubuntu) # Identify network interface name ip addr # Note: interface naming varies — ens33, eth0, ens160, etc. # IMPORTANT: Do NOT run tcpdump from the Commvault base folder # The PCAP file can grow very large. Run from a filesystem with free space. df -h # Find a partition with at least 2-5 GB free (e.g., /var, /tmp) # Navigate to free space location cd /var # Run tcpdump — capture on backup interface, save to PCAP file # Syntax: tcpdump -i <interface> -w <output_file.pcap> tcpdump -i ens33 -w ma108_to_cs103_capture.pcap # Press Ctrl+C to stop capture when done # Fix permissions before copying chmod 755 ma108_to_cs103_capture.pcap # Copy PCAP to Windows machine for analysis using WinSCP # Connect WinSCP to Linux MA → navigate to /var → copy .pcap to desktop # On Windows CommServe (where Wireshark is installed): # Open Wireshark → File → Open → select the .pcap file # Apply filter: ip.addr == 192.168.1.103 # Look for dark rows → TCP RST entries
💡 Analyzing Linux PCAP in Wireshark

TCPDump generates .pcap files in the same format as Wireshark captures. You can open them in Wireshark for the same GUI analysis. Simply copy the .pcap file from the Linux machine (via WinSCP or SCP) to any Windows machine with Wireshark installed, then open it with File → Open. The offline analysis process is identical to live capture analysis.

🔄
VSA Proxy Migration
Replacing or decommissioning an existing VSA access node with a new one

When VSA Migration Is Needed

There are common scenarios requiring a VSA (Virtual Server Agent) proxy to be replaced: the existing server is being decommissioned, you are migrating from Windows to Linux access node to reduce licensing costs, the current VM is undersized and needs to be replaced with a larger one, or the OS needs upgrading to a newer certified version.

📘 Credentials Are NOT Stored on the VSA

A common concern when decommissioning a VSA is: "What happens to the credentials stored on it?" The answer is — credentials are stored in the Commvault CommServe database, not on the VSA proxy. You can verify this in Control Panel → Credential Vault — all authentication credentials (cloud managed identities, service accounts, etc.) are stored in the central database. The VSA is just a compute proxy — it holds no sensitive configuration data that would be lost on decommission.

VSA Migration Steps — Command Center

🔄 VSA Proxy Migration — Before & After
Swapping the access node while maintaining all configuration, credentials, and backup history
Before Migration
☁ Hypervisor (GCP / Azure / VMware)
↓
🔴 Old VSA Proxy (decommission target)
↓
💾 Cloud Library / Storage
SWAP NODE →
After Migration
☁ Hypervisor (same — config updated)
↓
🟢 New VSA Proxy (same packages + settings)
↓
💾 Cloud Library / Storage (unchanged)
✅ Credentials → stored in CommServe DB (not on VSA)
✅ Job History → stored in CommServe DB (not on VSA)
✅ No data migration needed — VSA is stateless
Replacing a VSA Access Node (Azure, GCP, VMware, or any hypervisor)
1
Deploy and Prepare the New Access Node VM

Create the replacement VM in your cloud or on-premises environment. Verify the OS is in Commvault's supported list for the specific workload (e.g., for Azure VM backup: Windows Server 2019, Oracle Linux, Red Hat Linux). Check the System Requirements Guide in Commvault documentation for your version.

2
Open Required Ports on the New VM

Ensure ports 8400 and 8403 are open bidirectionally. Port 22 (Linux) or 135/139/445 (Windows) must be open temporarily for push installation. For cloud VMs: add NSG/VPC/Security Group rules before attempting install. For cloud connectivity (e.g., Azure/GCP): ensure the VM has network access to the cloud management endpoints.

3
Install the Same Packages as the Old VSA

Check what packages are installed on the old VSA: CommCell Console → Client Computers → right-click old VSA → View Installed Software. Install exactly the same packages on the new VM via push install (Tools → Add/Remove Software → Install). For Azure VM backup: Media Agent + Virtual Server Agent + IntelliSnap + Storage Accelerator. For GCP VM backup: Virtual Server Agent + Storage Accelerator.

4
Apply Any Additional Settings

If Commvault Support has previously applied any registry keys, additional settings, or environment-specific customizations on the old VSA, replicate those on the new one. Check the old VSA's additional settings in CommCell Console → Client → Properties → Additional Settings. Copy all custom entries to the new VM.

5
Edit Hypervisor — Swap the Access Node

Command Center → Protect → Virtualization → select the Hypervisor → Configuration → Edit. Uncheck the old VSA access node and add the new one. Click Save. The hypervisor is now using the new VSA for all backup and restore operations.

6
Run a Test Backup and Confirm

Trigger a manual backup job using the new VSA. Confirm it completes successfully. Monitor the Job Controller — if it runs as expected, the migration is complete. Keep the old VSA online for a few days as a rollback option. Only delete the old VSA from Commvault after confirming multiple successful backup cycles on the new node.

⭐ Keep the Old VSA as a Rollback Buffer

Do not immediately delete the old VSA from Commvault after the migration. Keep it available for at least a few days. If any unexpected issues arise with the new VSA, you can quickly revert by editing the hypervisor configuration and switching back. Once you confirm multiple successful backup cycles on the new VSA, it is safe to remove the old one.

Dual Interface / Backup Network Configuration

For environments with multiple network interfaces on the access node (e.g., one NIC for backup traffic, another for replication/auxiliary copy), you can configure which IP Commvault uses for which operation.

Location in Commvault Control Panel → Backup Network. Configure network pairs here — select which IP to use for backup and which for auxiliary copy/replication. Control Panel
How It Works Define an interface pair: this IP is for backup, that IP is for replication. Commvault auto-detects available IPs. All agents using that access node inherit the same network configuration. Auto-detected
Prerequisite Both IPs must have network connectivity to their respective destinations. Without connectivity, the jobs will fail regardless of what is configured in Commvault. Connectivity Required
☁
GCP VM Backup — Overview & Architecture
Why customers use Commvault alongside Google Cloud's native backup tools

Cloud Coverage — Why GCP?

The majority of enterprise customers worldwide use one of three cloud providers: AWS, Azure, or Google Cloud Platform (GCP). Commvault has strong compatibility with all three. While each cloud has native backup tools, customers prefer Commvault because it provides:

  • Unified reporting: A single dashboard for backups across on-premises, AWS, Azure, and GCP
  • Cross-cloud restore: Restore a GCP VM to Azure or on-premises — not possible with native GCP tools
  • Granular retention control: Define custom retention per workload, unlike GCP's preset options
  • Enterprise compliance: Required by many regulatory frameworks to have third-party backup verification
📘 Agentless VM Backup — Same Principle Across Clouds

GCP VM backup in Commvault is agentless — just like AWS EC2 and Azure VM backup. You do not install any Commvault agent inside the production VM. Instead, Commvault uses an external access node (VSA proxy) running in GCP to orchestrate snapshot-based backups via the GCP API. The production VM can run any OS — Linux, Windows, or otherwise.

High-Level Architecture

☁ GCP VM Backup — End-to-End Architecture
Complete data flow from production VMs through the access node to cloud library storage
On-Premises / Management Layer
🏗 CommServe (On-Prem) Job Controller & Scheduling
Command Center (Cloud Config)
Credential Vault (JSON keys stored here)
CommServe DB (job history, catalog)
Port 8400 / 8403 ↕
GOOGLE CLOUD PLATFORM
🖥 Production VMs Ubuntu / RHEL / Windows
No agent installed (agentless)
Customer workloads & data
GCP API
snapshot
→
🔄 GCP Access Node (VSA) Rocky Linux 8 / RHEL / Windows
Packages: VSA + Storage Accelerator
+ Cloud Apps (for object storage)
Orchestrates snapshots via GCP API
backup
data
→
🪣 GCP Cloud Library Cloud Storage Bucket
Nearline storage class
Same region as VMs
Backup data target
🔑 Service Account (VM Backup) Owner Role
JSON key file auth
Used by Hypervisor config
🔑 Service Account (Storage) Storage Admin Role
JSON key file auth
Used by Cloud Library config
🛡 VPC Firewall Rules Ingress: 8400, 8403 (permanent)
Ingress: 22 (temp — push install)
Egress: open by default
Production VMsAny OS (Ubuntu, RHEL, Windows)
No Commvault agent
Agentless backup
→
GCP Access Node (VSA)Rocky Linux / RHEL / Windows
VSA + Storage Accelerator packages
Deployed in GCP
→
GCP Cloud LibraryGoogle Cloud Storage bucket
Nearline storage class
Same region as VMs
GCP Service AccountJSON key file
Owner or custom role
Authenticates Commvault to GCP API
+
CommServe (on-prem)Commvault management server
Command Center configuration
Job Controller & reporting
📘 GCP Terminology vs Azure/AWS

GCP uses different terminology than Azure and AWS. Project = Azure Subscription / AWS Account (billing unit). Instance = Virtual Machine. Bucket = Azure Blob Container / AWS S3 Bucket. Service Account = Azure Managed Identity / AWS IAM Role. Cloud Storage = Azure Blob Storage / AWS S3. VPC Firewall Rules = Azure NSG / AWS Security Groups.

Prerequisites Checklist — GCP VM Backup

Step 1 — List VMs to Protect Identify which GCP VM instances need agentless backup. Discuss with the customer — any OS is supported since backup is agentless from Commvault's side. Planning
Step 2 — Deploy GCP Access Node Create a GCP VM instance for the access node. Supported OS: Rocky Linux 8, Red Hat Linux, or Windows Server. Deploy in the same GCP region as the production VMs. Rocky Linux / RHEL / Windows Only
Step 3 — Open Firewall Ports GCP VPC Firewall Rules: create ingress rules for ports 8400 and 8403 (permanent, all instances in network). Port 22 (Linux) for push install — temporary only. Outbound is open by default in GCP. VPC Firewall Required
Step 4 — Install VSA Packages Push install from CommServe to access node: Virtual Server Agent + Storage Accelerator. Storage Accelerator is needed so the VSA can write directly to the GCP Cloud Library without a separate media agent. Both Packages Mandatory
Step 5 — Create GCP Service Account GCP Console → IAM → Service Accounts → Create. Assign owner role (or custom role). Download JSON key file — this is the authentication credential for Commvault. JSON File Required
Step 6 — Create Cloud Storage Bucket GCP Console → Cloud Storage → Create Bucket. Select the same region as VMs. Recommended storage class: Nearline (best for backup/restore workloads). Standard is for production; Coldline/Archive for long-term retention. Nearline Recommended
Step 7 — Configure in Command Center Cloud hypervisors (GCP, Azure, AWS) can only be configured from Command Center — the classic Java CommCell Console does not support this. Create cloud library, plan, and GCP hypervisor in Command Center. Command Center Only
🖥
GCP Lab Setup — Creating VMs and Users
Deploying production VM (Ubuntu) and access node (Rocky Linux) in Google Cloud

GCP Free Trial & VM Creation

Google Cloud offers $300 free credit for 3 months for new accounts — more generous than Azure's limited $200. Payment details are required at signup, but Google charges a temporary ₹1000 (for Indian accounts) that is refunded when the account is closed. The account will not be auto-charged after trial ends — you must manually upgrade to a paid account.

Production VM Spec e2-micro (1 vCPU, 1 GB RAM) — sufficient for demo/lab. OS: Ubuntu 22.04 LTS. Disk: 20 GB. No Commvault agent installed — this is the agentless backup target VM. Ubuntu — Any OS
Access Node VM Spec e2-standard-2 (2 vCPU, 4 GB RAM minimum — B2S equivalent). OS: Rocky Linux 8 (certified by Commvault). Disk: 30 GB+. Commvault VSA + Storage Accelerator will be installed here. Rocky Linux 8 Required
Region Deploy both VMs in the same region (e.g., asia-south1 for India). Keeping VMs and cloud library in the same region avoids GCP egress charges for data transfer. Same Region = No Egress
📘 GCP Instance vs VM Naming

GCP calls virtual machines "instances" — the term "VM instance" is used throughout the GCP console. Machine names must be in lowercase only (no uppercase letters allowed). When you click the SSH button on an instance in the GCP console, it opens a browser-based terminal directly connected to that VM — no separate SSH client or key pair setup required initially.

Creating VM Users (Linux — No Default Password)

GCP Linux VMs do not show a username/password creation screen during instance setup. You must create users manually after connecting via the browser console. The steps differ slightly between Ubuntu and Rocky Linux.

bash — Create User for SSH Access (Ubuntu / Rocky Linux)
# Connect via GCP Console → VM Instances → SSH button # This opens browser terminal — no username/password needed initially # Create a new user (Ubuntu uses 'adduser', Rocky/RHEL uses 'useradd') sudo adduser abdul # Ubuntu: adduser (interactive, asks for password) sudo useradd -m abdul # Rocky Linux / RHEL: useradd sudo passwd abdul # Set password for the user # Grant sudo (elevated/root) access # Ubuntu: add to 'sudo' group sudo usermod -aG sudo abdul # Rocky Linux / RHEL: add to 'wheel' group sudo usermod -aG wheel abdul # Verify elevated access su - abdul sudo whoami # Should output: root # Enable password authentication for SSH (needed for PuTTY/external SSH) # Edit SSH daemon config sudo vi /etc/ssh/sshd_config # Find and modify these lines: # PasswordAuthentication no → PasswordAuthentication yes # Ubuntu also requires: # KbdInteractiveAuthentication no → KbdInteractiveAuthentication yes # Restart SSH service after changes sudo systemctl restart sshd # Now connect from external PuTTY using the GCP instance's External IP # (External IP = Public IP, shown in GCP console VM instances list) # Note: External IP changes every time VM is stopped and started # In production, use static/reserved IP to avoid this
💡 Simulating Customer Data for Demo

On the production VM, create test directories and files to simulate customer data. After backup and restore, verify these files still exist — this proves the backup captured the data correctly. Example: mkdir -p /usr/customer_data then create F1, F2, F3 files inside. After full VM restore, SSH into the restored VM and verify the directory and files are intact.

🔑
GCP Service Account & JSON Authentication
Creating service accounts, assigning roles, and downloading JSON keys for Commvault

What is a GCP Service Account?

A GCP Service Account is the authentication mechanism for Commvault to interact with GCP APIs. It is equivalent to an Azure Managed Identity or AWS IAM Role. The service account is given specific permissions (roles), and Commvault authenticates using a JSON key file downloaded from that service account.

🔑 GCP Service Account Authentication Flow
How the JSON key file travels from GCP IAM to Commvault and authenticates API calls
👤 GCP IAM Console Create Service Account
Assign Role (Owner / Custom)
Generate JSON Key
download
JSON file
→
📄 JSON Key File Contains private key
project_id, client_email
One-time download only!
upload to
credential
→
🏗 CommServe DB Credential Vault stores JSON
Used by Command Center
Persists across VSA migrations
↓ authenticates
🔄 Access Node → GCP API Calls VM discovery · Snapshot creation · Disk mount/unmount
Bucket read/write · Resource cleanup
🔵 Owner Role (Highest Privileges)
  • Full access to all resources in the GCP project
  • Used for hypervisor (VM backup) configuration
  • Simplest to configure — no custom role creation needed
  • Not recommended for production — too broad
  • Acceptable for lab/test environments
🟣 Custom Role (Least Privilege)
  • Only the specific permissions Commvault needs
  • Used for cloud library (storage admin for bucket operations)
  • More secure — follows principle of least privilege
  • Requires creating a custom role in GCP IAM
  • Recommended for production environments
📘 Two Service Accounts — Two JSON Files

In the lab, two separate service accounts are created: (1) One with Owner role — used for configuring the GCP hypervisor (VM backup). (2) One with Storage Admin custom role — used for configuring the GCP Cloud Library (bucket access). You can use a single Owner service account for both, but creating separate accounts demonstrates best practices and how to customize permissions per use case.

🔐 Cross-Cloud Authentication Comparison
How Commvault authenticates to each major cloud provider
🔵 Azure Managed Identity
System-assigned · No keys
Tenant ID + Subscription ID
Custom Role assignment
🟠 AWS IAM Role
Access Key + Secret Key
Instance Profile (EC2)
ARN-based permissions
🟢 GCP Service Account
JSON key file download
Project-scoped permissions
Owner or Custom Role

Creating a Service Account and Downloading JSON Key

GCP Console — Service Account Setup for Commvault
1
Enable the Cloud Resource Manager API

GCP Console → search for "Cloud Resource Manager API" → Enable. This API allows third-party tools like Commvault to interact with GCP project resources. Without it, the JSON key download may not work and Commvault will be unable to discover resources.

2
Create the Service Account

GCP Console → IAM & Admin → Service Accounts → Create Service Account. Provide a meaningful name (e.g., commvault-vm-backup or commvault-cloud-library). The Service Account ID is auto-generated — it becomes the email-like identifier (e.g., [email protected]).

3
Assign a Role to the Service Account

During creation, or via IAM → Grant Access: for VM backup, assign the Owner role at project scope. For storage/library access, create a custom role with Storage Admin permissions (IAM → Roles → Create Role → filter by storage admin → add all storage permissions → save). Assign the custom role to the storage service account.

4
Download the JSON Key File

IAM → Service Accounts → click on the service account → Keys tab → Add Key → Create New Key → select JSON format → Create. A JSON file downloads immediately. Store it securely — it cannot be re-downloaded and grants access to your GCP project. This file is uploaded into Commvault as the authentication credential.

⚠ JSON File is a One-Time Download

Once you close the download dialog, the JSON key cannot be downloaded again from GCP. If you lose it, you must generate a new key (Keys tab → Add Key → Create New Key). Treat the JSON file like a password — store it in a secure location (password manager, secrets vault). Anyone with this file can access your GCP project resources with the permissions granted to that service account.

What's Inside the JSON Key File

JSON — GCP Service Account Key Structure
{ "type": "service_account", "project_id": "my-first-project-123456", "private_key_id": "abc123def456...", "private_key": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----\n", "client_email": "commvault-vm-backup@my-first-project-123456.iam.gserviceaccount.com", "client_id": "123456789012345678901", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/..." } // Key fields to identify which service account the JSON belongs to: // "project_id" — your GCP project identifier // "client_email" — the service account ID (email format) // The private_key is the actual authentication secret
🪣
GCP Cloud Library Configuration
Creating the Cloud Storage bucket and configuring it as a Commvault cloud library

Create the GCP Cloud Storage Bucket

A GCP Cloud Storage bucket is the object storage container that Commvault uses as its cloud library. Backup data is written here by the access node via the Storage Accelerator package.

Bucket Naming Bucket names must be globally unique across all GCP projects worldwide. Choose a meaningful name (e.g., commvault-cloud-storage-prod). Cannot be changed after creation. Must Be Globally Unique
Location Type Select Region (single region) for lab/cost savings. Multi-Region provides geo-redundancy but costs more. Select the same region as your access node and production VMs (e.g., asia-south1). Same Region as VMs
Storage Class Nearline is recommended for Commvault backups — optimized for data accessed less than once per month (infrequent access, good for backup/restore). Standard is for frequently accessed data. Coldline/Archive for long-term archival only. Nearline for Backups
Public Access Leave public access prevention enabled (default). Commvault connects via API with the service account credentials — no public internet exposure needed for the bucket. Keep Private

Bucket Permissions — Critical Lesson

🛡 GCP IAM Permission Inheritance — The Bucket Gotcha
Why project-level permissions may not automatically reach the bucket
🌐 GCP Project Level Service Account assigned
Storage Admin role here
Project-scope permissions
↓
May inherit
(not guaranteed)
↓ ✗
Often DOES NOT inherit
(causes "Access denied")
🪣 Bucket A (inherits ✅) Permissions inherited
from project level
Works by default
🪣 Bucket B (blocked ❌) Permissions NOT inherited
"storage.objects.list denied"
Must grant explicitly!
↓ FIX
✅ Explicit Bucket-Level Grant Bucket → Permissions → Grant Access
Enter Service Account email → Assign Storage Admin
Guaranteed to work — always do this!
⚠ Bucket Does NOT Automatically Inherit Project-Level Permissions

A critical discovery from the lab: even if you create a service account at the project level and assign it the Storage Admin role, the GCP bucket will not automatically inherit those permissions. If you configure the cloud library in Commvault without explicitly granting the service account access to the bucket, you will get: "Access denied. storage.objects.list permission denied on resource." Fix: go to the bucket → Permissions → Grant Access → enter the service account email → assign the custom storage role. After this explicit grant, library configuration succeeds.

✅ Service Account Created at Project Level
  • Service account has role assigned at project scope
  • New buckets may or may not inherit — depends on GCP IAM inheritance settings
  • Always verify bucket-level permissions explicitly
  • For safety: always add service account to bucket permissions directly
✅ Explicit Bucket-Level Permission Grant
  • Bucket → Permissions → Grant Access
  • Enter service account email (the "principal")
  • Assign the custom storage role (or Storage Admin)
  • Guaranteed to work — explicit trumps inherited

Configure Cloud Library in Commvault Command Center

Adding GCP Cloud Library in Command Center
1
Navigate to Storage Configuration

Command Center → Storage → Cloud → Add. Select Google Cloud as the cloud type. Give the library a meaningful name (e.g., gcplib).

2
Select Media Agent

Select the CommServe as the media agent (if it doubles as MA in lab), or select a dedicated media agent. In GCP VM backup lab, the access node acts as the data mover due to the Storage Accelerator package.

3
Configure Authentication (Service Account Method)

Authentication method: Service Account. Click the Credentials field → Add Credential → give a name → upload the JSON file (the storage admin service account JSON). Click Save. The credential is stored in the Commvault database — not on the access node.

4
Enter Bucket Details

Region: select the GCP region where the bucket was created (e.g., asia-south1). Storage Class: Nearline. Service Host: leave as default (storage.googleapis.com — same for all GCP public cloud). Bucket Name: copy from GCP Console exactly.

5
Save and Verify

Click Save. If permissions are correct, the library is created successfully. If you see "Access denied" or "Failed to verify device" — go back to GCP Console and explicitly add the service account to the bucket's permissions as described above. After fixing permissions, retry.

🔄
GCP VM Backup & Restore
Configuring hypervisor, running backup, and restoring a deleted GCP VM instance

GCP Snapshot Backup Process — Unique Differences vs Azure/AWS

GCP VM backup in Commvault works differently from Azure and AWS VM backups at the snapshot level. Understanding this difference is important for interviews and troubleshooting.

📸 GCP vs Azure/AWS — Snapshot Backup Process Flow
Key architectural difference in how snapshot data reaches the VSA proxy
Azure / AWS / VMware
1 Snapshot of VM disk created
↓
2 Snapshot directly mounted on VSA proxy
↓
3 VSA reads data → writes to library
↓
4 Snapshot deleted after backup
VS
GCP (& OpenStack)
1 Snapshot of VM disk created (gx-backup-*)
↓
2 New volume (disk) created from snapshot
↓ (snapshot deleted now)
3 Volume mounted on VSA → read data → write to library
↓
4 Temp volume deleted via GCP API after backup
⚠ GCP temp disk is visible in GCP Console → Disks, but NOT visible inside proxy OS via lsblk
Step Azure / AWS / VMware GCP (Google Cloud)
1. Snapshot Snapshot of VM disk created Snapshot of VM disk created (naming: gx-backup-<jobid>)
2. Mount Snapshot is directly mounted on the VSA proxy A new volume (disk) is created from the snapshot, then that volume is mounted on the VSA proxy
3. Snapshot cleanup Snapshot deleted after backup copy Snapshot deleted as soon as the volume is created from it
4. Read VSA reads data from mounted snapshot VSA reads data from the mounted volume (read-only)
5. Data visible inside OS? Visible via lsblk on proxy NOT visible inside proxy OS via lsblk — only visible in GCP Console as "Additional disk in use by access node"
6. Cleanup Snapshot deleted by Commvault after backup Volume deleted by Commvault API request after backup completes
📘 GCP Snapshot Naming Convention

Commvault-created snapshots in GCP follow the naming pattern: gx-backup-<jobid>. If you see a snapshot or disk in GCP Console with a name starting with gx-backup-, it was created by Commvault. The snapshot is created, then a disk/volume is generated from it. The temporary disk (same size as the production volume) is read-only mounted on the access node during backup. Once the backup job completes, Commvault deletes both the temporary disk and the original snapshot via GCP API. OpenStack also uses this same volume-from-snapshot method — it differs from Azure/AWS/VMware where the snapshot is mounted directly.

Configuring the GCP Hypervisor in Command Center

GCP Hypervisor Setup — Command Center Guided Setup
1
Open Hypervisor Configuration Wizard

Command Center → Protect → Virtualization → Add Hypervisor → select Google Cloud Platform. Alternatively: Guided Setup → Virtualization → Google Cloud. Cloud hypervisor configuration is available only in Command Center — not in the Java CommCell Console.

2
Provide Server Name and Credentials

Server Name: any user-defined name. Credentials: click Add → enter credential name → upload the JSON file for the hypervisor service account (owner role JSON, not the storage JSON). Click Save credential.

3
Select Access Node

Select the GCP access node (Rocky Linux VM with VSA + Storage Accelerator installed). Click Next. If the configuration fails here with "access node not reachable" — check if the access node VM is powered on and if the IP has changed since the last startup. Update the client hostname/IP in CommCell Console → Client Properties if needed.

4
Define VM Group Content

Browse by project or by VM name. Select the production VMs to protect (e.g., prod-vm). Remove the default "all unprotected VMs" content — only add the specific VMs you want to back up to avoid unintended VMs being included. Select the GCP storage plan.

5
Troubleshoot: Access Node IP Change

GCP external IPs change when VMs are stopped and started (unless reserved/static IP is configured). If the access node shows "Not Ready" after restart — update the IP: CommCell Console → Client Computers → right-click access node → Properties → update IP/hostname to the new external IP → Submit. Run Check Readiness again to confirm connectivity.

💡 Log File for Hypervisor Configuration Failures

If the hypervisor configuration fails or the access node shows issues, the primary log to check is CVD.log on the access node. This log captures all communication daemon events including connectivity errors, authentication failures, and configuration issues. It is the first log to check for any client-side connectivity problem.

Running Backup and Full VM Restore

🔄 GCP VM Backup → Delete → Restore Lifecycle
End-to-end demo workflow showing disaster simulation and recovery
1
Full Backup Snapshot → volume → read data → write to cloud library
→
2
Verify Job Controller: "Completed" status
→
3
Delete VM GCP Console → VM Instances → Delete (simulate disaster)
→
4
In-Place Restore Recovery Points → Full Instance Restore → Power On
→
5
Validate SSH into restored VM → verify customer data files intact
GCP VM Backup → Delete VM → Restore Demo
1
Run Full Backup

VM Group → right-click → Backup → Full. Monitor in Job Controller. Watch GCP Console → Snapshots during backup — a gx-backup-* snapshot appears briefly, then a temporary disk is created from it. The snapshot is deleted. The temporary disk shows as "In use by access node" in GCP Console → Disks. Once backup completes, the temporary disk is also deleted.

2
Verify Backup Then Delete the VM

Confirm backup completed successfully (Job Controller: Completed). Go to GCP Console → VM Instances → select prod-vm → Delete. Confirm deletion. The VM no longer exists in GCP. This simulates accidental deletion or disaster.

3
Restore In-Place from Commvault

Command Center → Protect → Virtualization → VM Group → select the VM → Recovery Points → select the latest recovery point → Full Instance Restore. Choose In-Place. Select options: Power on after restore and Unconditionally overwrite. Submit restore job.

4
Monitor Restore and Verify

Monitor the restore job in the Job Controller. After completion, go to GCP Console → VM Instances — the VM reappears with its original name and configuration. Verify: SSH into the restored VM → check that the test files (F1, F2, F3 in /usr/customer_data/) still exist. This confirms the backup was complete and the restore was successful.

🪣
GCP Object Storage (Cloud Storage Bucket) Backup & Restore
Backing up GCP Cloud Storage buckets using the Cloud Apps package and service account authentication

What is GCP Object Storage Backup?

In addition to backing up GCP VMs, Commvault can also back up the contents of GCP Cloud Storage buckets (object storage). This is useful when customers store critical application data or files in GCP buckets that need protection beyond what GCP's native versioning provides.

🪣 GCP Object Storage Backup — Architecture
Cloud Apps package enables bucket-level backup separate from VM backup
🪣 Source: GCP Bucket Customer data bucket
Objects / files / blobs
Any region in GCP
GCP Storage API
→
read objects
🔄 Access Node Packages installed:
• VSA (VM backup)
• Storage Accelerator
• Cloud Apps (object storage)
write backup
→
data
💾 Target: Cloud Library Separate GCP bucket
Nearline storage class
Backup repository
Auth Method 1 Service Account + JSON file ✅ recommended
Auth Method 2 Access Key + Secret Key (needs default project set)
Config Path Guided Setup → Object Storage → Google Cloud
Required Package The Cloud Apps package must be installed on the access node (in addition to VSA and Storage Accelerator). Object storage backup uses a different package from VM backup. Cloud Apps Package
Same Access Node You can use the same GCP access node for both VM backup (VSA package) and object storage backup (Cloud Apps package). Install both packages on the same VM to avoid provisioning separate machines. Reuse Access Node
Configuration Path Cannot always be found via Protect → Object Storage in Command Center (UI varies by MR version). Use Guided Setup → Object Storage instead. Select Google Cloud as the vendor. Guided Setup Recommended
Authentication Methods Two methods: (1) Access Key + Secret Access Key — generated from GCP Console → Cloud Storage → Settings → Interoperability tab. (2) Service Account + JSON file — same JSON file approach used for VM backup. Both work; Service Account is more secure. Two Methods Available

Access Key Authentication — Important Scope Setting

⚠ Access Keys Require a Default Project to Be Set

A critical lab discovery: configuring object storage backup using Access Key + Secret Access Key initially failed with "credentials incorrect." The keys were valid, but GCP requires a default project to be associated with the access keys. This is done in GCP Console → Cloud Storage → Settings → Interoperability → define the default project for the user account. Once the project scope was defined, the same keys worked immediately. This is a non-obvious GCP requirement that is easy to miss.

Service Account Authentication — Object Storage

The preferred method. Uses a JSON file from a service account with Storage Admin permissions — the same approach as cloud library authentication.

Configuring GCP Object Storage Backup via Guided Setup
1
Install Cloud Apps Package on Access Node

Command Center → the access node client → three-dot menu → Add Software → search for Cloud Apps → Install. Monitor via install.log on the access node. After installation, confirm the access node shows "Ready" status. Log files to monitor: install.log and updateinfo.log on the access node.

2
Create Service Account for Object Storage

GCP Console → IAM → Service Accounts → Create (e.g., gcp-storage-backup-sa). Assign Storage Admin role (or custom storage role). Download the JSON key file. This can be the same service account used for the cloud library, or a separate dedicated one.

3
Create the Production Bucket with Test Data

GCP Console → Cloud Storage → Create Bucket (e.g., gcp-customer-data-bucket). Upload test files inside the bucket to simulate customer data. Verify the service account has permissions on this bucket (either inherited from project-level role, or explicitly granted at bucket level).

4
Configure Object Storage in Commvault Guided Setup

Command Center → Guided Setup → Object Storage → Google Cloud → Next. Select the backup plan (same GCP storage plan). Select the access node (where Cloud Apps is installed). Authentication: select Service Account → Upload JSON file → enter Project ID (found in GCP Console → Cloud Overview → Project ID). Click Next to validate credentials.

5
Select Bucket Content and Finish

Browse the dropdown — it shows available buckets in the project. Select the production bucket. Click Finish. The object storage instance is now configured. Run a full backup, then delete the bucket to simulate data loss, then restore in-place to verify the bucket and its contents are recovered.

💡 Bucket in Different Region from Access Node — Does It Work?

In the lab, a bucket was accidentally created in US-East-1 while the access node and other resources were in Asia-South-1 (Mumbai). The backup still succeeded — Commvault can access GCP buckets in any region from any access node, provided the service account has the correct permissions. However, cross-region access incurs GCP egress charges. Best practice: always create the production bucket in the same region as the access node to avoid egress costs.

⭐
Best Practices — Network, GCP, and Cloud Backup
Production guidance for network diagnostics, GCP deployments, and cloud library design

Network Troubleshooting Best Practices

Always Work in Layers Start with ping (ICMP) → netstat (local listening) → CVPing/telnet (port reachability) → traceroute (routing path) → CV Network Test Tool (throughput) → Wireshark (packet-level). Do not jump straight to Wireshark. Bottom-Up Approach
Run Multiple Iterations Never rely on a single CV Network Test Tool result. Run 2-3 iterations in both directions. Discard outliers and use the consistent result as the reported throughput. Multiple Runs Required
Verify IP Before Escalating Always double-check destination IPs with the target machine directly (ipconfig / ip addr). Human IP entry errors are extremely common and will waste time escalating a non-existent network problem. Verify IP First
Wireshark — Always Take Downtime Never run Wireshark on a production server without scheduling a downtime window and having the Windows/OS team on-call. A degraded server can hang when Wireshark is started due to additional load. Downtime Required
TCPDump File Location Never run tcpdump from the Commvault base folder. PCAP files grow very large — run from a partition with adequate free space (check with df -h first). Avoid Base Folder
⭐ Your Role Ends at Diagnosis

As a backup admin, your responsibility is to diagnose the connectivity issue and present evidence. It is the network team's responsibility to fix MTU mismatches, replace faulty hardware, update switch configurations, or resolve TCP reset issues. Use the tools to generate the proof, present it clearly, and formally escalate to the correct team. Do not attempt to modify network infrastructure yourself unless you own that domain.

GCP Backup Best Practices

Same Region for All Components Deploy access node, production VMs, and cloud storage bucket all in the same GCP region. Cross-region data transfer incurs significant GCP egress charges and increases latency. No Egress Charges
Static External IP for Access Node Reserve a static external IP for the GCP access node VM. By default, GCP assigns a new external IP every time the VM is stopped and started, requiring manual IP updates in Commvault every restart. Use Static IP
Service Account Naming Conventions Use meaningful names — never personal names. Use role-based names like commvault-vm-backup-sa, commvault-storage-sa. Makes IAM audits and team handovers much cleaner. Meaningful Names
Nearline for Backup Storage Class Always use Nearline storage class for backup libraries — not Standard (expensive for infrequent access) and not Coldline/Archive (high retrieval cost for frequent restores). Nearline provides the best cost/performance balance for backup and restore workloads. Nearline Always
Stop VMs When Not in Use (Lab Only) For lab environments, always stop GCP VMs when not in use — billing continues for running instances. In production, VMs must stay running for backup jobs to execute on schedule. Lab Cost Saving
Secure the JSON Key File Store JSON key files in a password manager or secrets vault. Rotate keys periodically. If a JSON file is compromised, revoke it immediately in GCP Console → Service Account → Keys → Delete key, then generate a new one. Secure JSON Files

GCP $300 Free Trial — Maximize for Learning

☁ GCP Free Trial vs Other Clouds

GCP provides $300 of free credit for 3 months with no VM type restrictions — you can deploy any VM instance type including the larger ones needed for access nodes. By comparison, Azure provides $200 for 30 days with some resource limits, and AWS does not provide a general free credit for new accounts. For Commvault training and lab purposes, GCP's free trial is the most flexible option. The temporary ₹1000 (India) charged at signup is returned when the account is closed — no surprise charges occur after trial expiry unless you manually upgrade.

🎓
Interview & Exam — Scenario-Based Q&A
Real-world questions on network troubleshooting, VSA migration, and GCP backup
A backup admin suspects a network issue between the CommServe and Media Agent. The network team says "everything is fine." What steps do you take to diagnose and prove there is a network problem?
Start with layered diagnostics: (1) Ping — basic ICMP reachability check. Note: may be disabled at customer sites. (2) Netstat — verify port 8400 is in LISTENING state on the media agent. (3) CVPing — test port 8400/8403 connectivity from CommServe to MA. CVPing ships with every Commvault agent and works even when telnet is blocked. (4) Check Readiness in CommCell Console — quick pass/fail port check. (5) CV Network Test Tool — measure throughput between the two entities in both directions. If below 50 MB/s, this is evidence of bandwidth degradation. (6) Wireshark / TCPDump — packet capture to find TCP connection resets (not retransmissions, which are normal). Present the captured TCP resets to the network team — this is the definitive proof that forces them to investigate. Common root causes include MTU mismatch, degraded NIC hardware, faulty cables, or switch port issues.
You run CVPing from CommServe to Media Agent and it times out. Services are running, Windows Defender is off, and NSG allows 8400/8403. What else could be wrong?
The most common overlooked cause in this scenario is a wrong IP address — a human entry error. Verify the destination IP by running ipconfig directly on the media agent and compare to what was entered in CVPing. A single digit difference (e.g., 160 vs 110) produces the same timeout. Additionally check: (1) Corporate/network firewall between the two servers — NSG open does not mean the network firewall is open. Raise a firewall change request with the network team. (2) For cloud VMs with multiple IPs — ensure the backup IP is being used, not a management or storage IP. (3) Run netstat -enob | findstr 8400 on the media agent to confirm the port is actually in LISTENING state — services appearing "running" doesn't guarantee the port is bound.
What is the CV Network Test Tool used for, and how do you run it in a restricted environment where port 25000 is blocked?
The CV Network Test Tool measures network throughput (bandwidth in MB/s) between two Commvault entities. It is used when backups are running slowly and a bandwidth bottleneck is suspected — the results provide concrete data to share with the network team. Normally it uses port 25000. In restricted environments where only 8400/8403 are allowed: schedule a maintenance window, stop all Commvault services on both machines (bringing them to a maintenance state), then run the CV Network Test Tool and manually specify port 8400 (or 8403) instead of the default 25000. After the test completes, restart the Commvault services on both machines. This is disruptive but necessary when port 25000 is blocked by security policy. On Linux, the tool is CLI-only and case-sensitive: ./cvnetworktesttool from the base folder.
You need to replace an existing VSA access node with a new VM. What are the key steps and what data needs to be migrated from the old VSA to the new one?
VSA migration is straightforward. Key steps: (1) Deploy the new VM with a supported OS (verify in Commvault documentation System Requirements). (2) Open ports 8400/8403 permanently and port 22 (Linux) temporarily for push install. (3) Check what packages are installed on the old VSA via CommCell Console → right-click client → View Installed Software. Install the exact same packages on the new VM via push install. (4) Copy any additional settings (registry keys, environment-specific settings added by Commvault Support) from old VSA's Additional Settings to the new one. (5) Edit the hypervisor in Command Center → uncheck old access node, add new one → Save. (6) Run a test backup; keep the old VSA live as rollback for a few days before decommissioning. Regarding data migration: no data needs to be migrated. Credentials are stored in the Commvault CommServe database (Credential Vault), not on the VSA. Backup job history and catalog are in the CommServe DB. The VSA is stateless compute only.
What is a GCP Service Account, and how does it differ from Azure Managed Identity and AWS IAM? How is it used in Commvault?
A GCP Service Account is a non-human identity that represents an application or service and is used for API authentication. It is the GCP equivalent of Azure Managed Identity (for cloud-native auth) or AWS IAM Role (for programmatic access). The key difference is authentication mechanism: Azure Managed Identity uses a system-assigned identity with no keys to manage. AWS IAM uses access key + secret access key or instance profiles. GCP Service Account uses a JSON key file — a downloaded private key that must be stored securely and uploaded to Commvault as a credential. In Commvault, the JSON file is uploaded as a credential in Command Center or the credential vault — it is stored in the CommServe database and used by the access node to authenticate API calls to GCP for VM discovery, snapshot creation, disk mounting, and cloud storage operations.
You configure a GCP Cloud Library in Commvault and get "Access denied — storage.objects.list permission denied on resource." The service account has Storage Admin role at project level. What is wrong?
This is a known GCP IAM inheritance issue. Even though the service account has Storage Admin role assigned at the project level, GCP Cloud Storage buckets do not always automatically inherit those project-level permissions. The bucket maintains its own permission list, which must be explicitly populated. Fix: go to GCP Console → Cloud Storage → select the bucket → Permissions tab → Grant Access → enter the service account email (principal) → assign the Storage Admin or custom storage role → Save. After this explicit bucket-level permission grant, Commvault can successfully verify the device and create the cloud library. This is a common pitfall for engineers migrating from Azure (where resource-level permissions inherit more transparently from subscriptions).
How does GCP VM backup in Commvault differ from Azure and AWS VM backup at the snapshot level?
The key difference is in how the snapshot data is presented to the VSA proxy. In Azure and AWS (and VMware), Commvault takes a snapshot of the VM disk and directly mounts the snapshot on the VSA proxy — the proxy reads backup data from the snapshot directly. In GCP (and OpenStack), Commvault takes a snapshot of the VM disk, then creates a new volume (disk) from that snapshot, and mounts the newly created volume on the VSA proxy. The temporary disk (same size as the production volume) is read-only mounted on the access node and is visible in GCP Console → Disks as "in use by access node" — but it does not appear inside the access node OS via lsblk or fdisk. Once the backup completes, Commvault deletes both the temporary disk and the original snapshot via GCP API. This extra volume-creation step is unique to GCP and OpenStack.
GCP object storage backup fails with "make sure credentials are correct and active." The access keys from the Interoperability tab are definitely correct. What is the likely cause?
This is a non-obvious GCP requirement: when using Access Key + Secret Access Key from GCP Cloud Storage's Interoperability settings, GCP requires a default project to be associated with those keys. Without setting a default project, GCP cannot determine the billing scope for the request, and authentication fails even if the keys are correct. Fix: GCP Console → Cloud Storage → Settings → Interoperability tab → look for "User Account HMAC keys" → set the default project for your user account. Once the project scope is defined, the same keys work immediately. Alternatively, switch to Service Account + JSON file authentication, which does not have this project scope requirement and is the recommended method for production environments.