Comment on page
Deploying on AWS with CloudFormation

Replicated 3 Node Cluster
Prerequisites
​AWS cli​
​Kubectl​
​jq​
zip
The CloudFormation script requires that the region in which Grainite will be deployed must have a minimum of 3 availability zones. For example, us-west-1 has only 2 available zones but us-east-2 has 3.
Tokens:
Helm deploy token (same as GitLab deploy token)
Helm username
Quay username
Quay password
Recommendation: All of the steps below should be performed from a linux VM running within the same virtual private cloud as the target cluster.
Download scripts
The scripts package contains scripts that make it easier to deploy and manage Grainite clusters with scripts for creating roles, VPCs, Grainite clusters, etc.
1.Run the following to download the CloudFormation AWS and Terraform GCP scripts package tar:
curl -H "DEPLOY-TOKEN: <token>" https://gitlab.com/api/v4/projects/26443204/packages/generic/scripts/<version>/cluster_creation.tar -o cluster_creation.tar
Replace <token> with the deploy token provided to you.

Also, replace <version> with the desired version of Grainite (e.g. 2316.1) that needs to be deployed or latest for the latest available version of Grainite.
2. Run the following to extract the script package tar:
tar xvf cluster_creation.tar
Setup and Create Roles
This step only needs to be performed once per region. Subsequent clusters can be created by reusing these roles.
1. Setup Roles
./grainite/scripts/bin/aws-grainite role-awsqs-create -h
​
role-awsqs-create: Create an AWSQS role to activate in the region to use
​
Usage:
    role-awsqs-create  STACK-NAME [flags]
​
Flags:
    -N:         dry run
    -h          help
Example:
./grainite/scripts/bin/aws-grainite role-awsqs-create tom-awstest
The command above should have automatically created and registered execution roles for AWSQS::EKS::CLUSTER and AWSQS::KUBERNETES::HELM in the same region where the role is created.  Please confirm the roles were created and are privately registered as shown below.
2. Create a VPC Proxy Role
./grainite/scripts/bin/aws-grainite role-vpcproxy-create -h
​
role-vpcproxy-create: Create a VPC proxy role for AWSQS to use
​
Usage:
    role-vpcproxy-create STACK-NAME [flags]
​
Flags:
    -N:         dry run
    -h          help
As the role that will be created through this command is global, it only needs to be run once.
Setup VPC
Create a dedicated VPC with 1 public subnet and NatGateway. This step can be skipped if the workload will be deployed using an existing VPC (with a NatGateway).
./aws-grainite vpc-create -h
​
vpc-create: Create a VPC
​
Usage:
    role-awsqs-create STACK-NAME [flags]
​
Flags:
    -R:         region
    -V:         environment name
    -s:         public subnet CIDR in the VPC
    -v:         VPC CIDR
    -N:         dry run
    -h          help
​
Example:
./grainite/scripts/bin/aws-grainite vpc-create tom-awstestvpc -V tom-awstestenv -s 172.31.20.0/24 -v 172.31.0.0/16 -R us-east-2 
Create an unsecured 3 node EKS cluster
Create a 3-node cluster in an existing VPC and deploy grainite workload using 3-node-with-vpc.yaml:
./grainite/scripts/bin/aws-grainite cluster-create -h
Creates a new unsecured grainite cluster with the given cluster name
​
Usage:
    cluster-create CLUSTER-NAME [flags]
​
Flags:
    -R:         region
​
    -i:         grainite image name
    -j:         jetty image name
    -r:         image repository URL
    -U:         image repository user name
    -T:         image repository access token
​
    -H:         helm repo URL
    -u:         helm repo user name
    -K:         helm repo access token
    -C:         helm chart version
​
    -S:         num servers in cluster, default: 3
    -n:         create the infrastructure without deploying Grainite workload
​
    -D:         type of persistent disks. default: gp3
​
    -m:         meta volume size suffixed with {Gi|Ti}, e.g. 1Ti
    -d:         dat volume size suffixed with {Gi|Ti} , e.g. 1Ti
​
    -c:         cloudformation template file name
    -v:         VPC stack name
    -V:         environment_name
    -k:         instance keypair name
    -s:         subnet1_cidr,subnet2_cidr,subnet3_cidr
    -x:         awsqs stack name
    -E:         use external LB for service end points with allowed source IP CIDR's, e.g., 20.21.22.0/24(,20.21.23.0/24)
    -N:         dry run
    -W:         wait for creation completion, true or false
​
    -h|--help   help
Where:
helm chart version : The release version for the helm chart.
Example: for release 2316.1, specify -C 23.16.1 or for release 2317, specify -C 23.17.0
environment_name: Environment tag
subnet_cidr: CIDR for 3 private subnets to be created in the VPC. These subnets should be already created.
awsqs_stack_name: The name of the stack used to create the role to activate the AWSQS extensions
instance key: Name of the ssh key pair to connect to worker nodes
template_file: The path and filename of the template to use to create the cluster. Default to 3-node-with-vpc.yaml
Example (Need to replace username, passwords and ssh key):
./grainite/scripts/bin/aws-grainite cluster-create tom-awstestgx2
    -v tom-awstestvpc
    -V tom-awstestenv
    -s 172.31.10.0/24,172.31.11.0/24,172.31.12.0/24
    -x tom-awstest
    -R us-east-2
    -C 23.16.1 
    -r quay.io/grainite
    -U QUAY-USERNAME
    -T QUAY-PASSWORD
    -i cluster:2316.1
    -H https://gitlab.com/api/v4/projects/26443204/packages/helm/stable
    -u HELM-USERNAME
    -K HELM-TOKEN
    -k SSH-KEY
    -c 3-node-with-vpc.yaml
Deploy a cluster with TLS and encryption enabled
Encryption can also be enabled on an existing cluster, see the following page for details:
Enabling Disk Encryption
Optionally, the following script can be used to create a cluster with encryption and TLS enabled directly:
Note: This will not create the client certificates necessary for TLS. To create these, follow Step 2 under Enabling TLS.
./grainite/scripts/bin/aws-grainite cluster-create-secure -h
​
Creates a new secure grainite cluster with the given name
​
Usage:
    cluster-create-secure CLUSTER-NAME [flags]
​
Env:
    GXS_CERTS_KEYS_DIR:
        Default GXS_CERTS_KEYS_DIR=${HOME}/.gxscerts/
​
        For secure setup of grainite cluster and enabling disk encryption export
        this env. The server CA CERTS and encryption KEYS will be generated under
​
            ${GXS_CERTS_KEYS_DIR}/<cluster-name>/{server/, client/, keys/}
​
Flags:
    -R:         region
    -e:         optional, setup disk encryption also
                keys will be generated under ${GXS_CERTS_KEYS_DIR}/<cluster-name>
​
    -i:         grainite image name
    -j:         jetty image name
    -r:         image repository URL
    -U:         image repository user name
    -T:         image repository access token
​
    -H:         helm repo URL
    -u:         helm repo user name
    -K:         helm repo access token
    -C:         helm chart version
​
    -S:         num servers in cluster, default: 3
​
    -D:         type of persistent disks. default: gp3
​
    -m:         meta volume size suffixed with {Gi|Ti}, e.g. 1Ti
    -d:         dat volume size suffixed with {Gi|Ti} , e.g. 1Ti
​
    -v:         VPC stack name
    -V:         environment_name
    -k:         instance keypair name
    -s:         subnet1_cidr,subnet2_cidr,subnet3_cidr
    -x:         awsqs stack name
    -E:         use external LB for service end points with allowed source IP CIDR's, e.g., 20.21.22.0/24(,20.21.23.0/24)
    -N:         dry run
​
    -h|--help   help
Where:
All flags are the same as those passed in Create a 3 node GKE Cluster except for -e which when passed enables encryption at desk.
Access the Kubernetes cluster
The Kubernetes cluster control plane and load balancer are private endpoints and can only be accessible from the same VPC and peered VPCs:
First, connect to the cluster by running the following command:
./grainite/scripts/bin/aws-grainite cluster-switch CLUSTER_NAME
Next, get the cluster's service endpoint by running the following command:
./grainite/scripts/bin/aws-grainite cluster-ip
Example:
$> ./grainite/scripts/bin/aws-grainite cluster-ip
a7486243db9f948a48faf52c07a69c31-d52a444e0cf31de2.elb.us-east-2.amazonaws.com
Destroy the cluster
Run the following command to destroy the cluster:
./grainite/scripts/bin/aws-grainite cluster-delete CLUSTER_NAME
Some of the subnets might not be deleted due to lingering lambda functions. To manually cleanup:
aws lambda list-functions --region <region-name> to find the function name for the subnets
aws lambda delete-function --function-name < function_name > --region <region-name>
Persistent volumes for statefulsets are also automatically removed with aws-grainite cluster-delete.
Optional: Clean up persistent volumes if cluster-delete fails
1. List persistent volumes used in the cluster
kubectl -n default get pv
2. Get the volume ID
aws ec2 describe-volumes --filter Name=status,Values=available Name=tag:cluster_name,Values=$CLUSTER_NAME --query “Volumes[*].VolumeId”3. Delete the volume
aws ec2 delete-volume --volume-id <volume_id>
Where:
<volume id> is the ID of the volume.
Administrators - Previous
Creating your Cluster
Deploying on Azure with Terraform
Last modified 4mo ago