How Do I Remove Nodes From Cluster In Vertica?

Before removing a node from a Vertica cluster, you must first remove it from the database. The procedure varies slightly depending on your deployment mode (Enterprise or Eon) and the tools you use (Administration Tools, Management Console, or admintools command line).

Prerequisites

• Back up your database: Always perform a full backup before making major cluster changes.
• Verify database state: Ensure all nodes in the database are in an UP or ACTIVE STANDBY state. If any node is down, start it before attempting to remove a node.
• Check K-safety: Confirm that removing a node will not cause the cluster to fall below the required K-safety value for high availability. If necessary, temporarily lower the K-safety level to proceed. You can check K-safety with the SELECT GET_COMPLIANCE_STATUS(); query.
• Access credentials: You will need dbadmin user credentials.

Method 1: Use Administration Tools (admintools)

This is a common method for Enterprise Mode databases, though it is also an option for Eon Mode.

1. Launch Administration Tools: Log in to a database host as the dbadmin user and run the following command:sh

   /opt/vertica/bin/admintools
   

   Use code with caution.

2. Navigate to Host Removal:

   • From the Main Menu, select Advanced Menu.
   • From the Advanced Menu, select Cluster Management.
   • From the Cluster Management menu, choose Remove Host(s) from Database.

3. Confirm the operation: The tool will issue a warning that you must redesign your database projections after removing hosts. Confirm the action to continue.

4. Select database and hosts:

   • Choose the database from which you are removing the node and click OK.
   • Select the hosts you want to remove and click OK.

5. Rebalance data: After the host is removed from the database, Vertica automatically initiates the rebalance process in Enterprise Mode to redistribute the removed node's data across the remaining nodes.

6. Remove the host from the cluster: After the host is removed from the database, exit Administration Tools. If the host is not used by any other database, you can remove it from the overall cluster configuration. Run update_vertica on a remaining host:sh

   /opt/vertica/sbin/update_vertica --remove-hosts host01,host02
   

   Use code with caution.

Method 2: Use Management Console (MC)

This graphical method is often preferred for Eon Mode databases, though it works for Enterprise as well.

1. Access the MC: Log in to your Vertica Management Console.
2. Navigate to database management:
   • Go to the Manage page for the cluster.
   • Click on the Databases tab.
3. Select the database: Choose the database from which you want to remove a node.
4. Remove the node:
   • In the Nodes section, find the node you want to decommission.
   • Click the Remove button associated with that node.
   • Confirm the removal when prompted.
5. Monitor rebalancing: For Enterprise Mode, Vertica will initiate an automatic rebalance. In Eon Mode, shards are rebalanced automatically when removing a primary subcluster node. You can monitor the progress on the Subclusters tab or via the REBALANCE_SHARDS function.
6. Remove the host from the cluster: Once the node is successfully removed from the database, use the admintools or update_vertica command line method to remove the host from the overall cluster configuration if it's no longer needed.

Method 3: Use the admintools command line

This is a scriptable, non-interactive method that is useful for automation.

1. Prepare the removal file: Create a file that lists the hosts you want to remove, for example, /tmp/remove_hosts.txt, with one hostname or IP address per line.``` host01.example.com host02.example.com

2. Remove nodes from the database: Run the db_remove_node command from a remaining cluster node. Vertica initiates an automatic rebalance after the removal.sh

   /opt/vertica/bin/admintools -t db_remove_node -d <database_name> -F /tmp/remove_hosts.txt
   

   Use code with caution.

   • -d <database_name>: Specifies the name of your database.
   • -F /tmp/remove_hosts.txt: Provides the list of hosts to remove.

3. Remove the host from the cluster: After the node is removed from the database, remove it from the cluster configuration using update_vertica:sh

   /opt/vertica/sbin/update_vertica --remove-hosts host01,host02
   

   Use code with caution.

Post-removal best practices

• Review system tables: After the removal and rebalancing are complete, query system tables such as NODES, PROJECTION_COLUMNS, and V_MONITOR.PROJECTION_STORAGE to verify the state and data distribution across the remaining nodes.
• Revisit K-safety: If you lowered K-safety, you can now raise it back to the desired level if enough nodes remain.
• Update projection designs: If removing the node significantly changed your cluster's topology, you may need to re-evaluate and re-create your projections for optimal performance.
• Decommission hardware: If the removed host is hardware, perform any necessary physical decommissioning or repurposing tasks.