Skip to main content

3 posts tagged with "deploy"

View All Tags

· 11 min read
Robert Barabas

This blog outlines the deployment of Tigris on an Google's Kubernetes Engine (GKE) Autopilot instance.

The installation will use recommended settings for redundancy, allocating more resources than a simple laptop based installation would. For more information on the laptop based installation please consult our previous blog!

If you would rather watch a video, check out the deployment in action on YouTube:

Requirements

Below are the requirements for the installation box and the target Kubernetes environment.

The list of items required:

  • Helm
  • Google Cloud SDK
  • git and tigris-deploy repository
  • GKE cluster with sufficient quotas

Installation Host

We will require Helm to perform the installation. It is assumed that the installation host already has access to the deployment target GKE cluster.

The version of helm used in this blog was:

❯ helm version
version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.19.2"}

To interface with the GKE cluster using kubectl conveniently, you may want to install the GKE plugin. You can install it with this command:

❯ gcloud components install gke-gcloud-auth-plugin

GKE

Fortunately, GKE Autopilot clusters automatically comes with a set of controllers installed. The list includes GKE Ingress that enables the creation of external load balancers for Ingress resources and controllers that manage other aspects of GCP, such as persistent disks.

One of the challenges of ensuring a successful deployment in GCP is to manage quotas efficiently. You will want to ensure quotas allow for sufficient CPU and SSD storage allocation.

Using the defaults of the Helm Chart, the following quotas proved to be sufficient:

GCP Quotas

Deployment

The installation deploys the following components:

  • Kubernetes Operator for FoundationDB
  • FoundationDB
  • Tigris Search (TypeSense)
  • Tigris Server

You can install the components individually or together, using the encompassing tigris-stack Helm Chart. Below I’m going to use this Chart to install Tigris.

Prepare For Deployment

Next, check out the deploy script repository:

git clone git@github.com:tigrisdata/tigris-deploy.git
Cloning into 'tigris-deploy'...
remote: Enumerating objects: 177, done.
remote: Counting objects: 100% (97/97), done.
remote: Compressing objects: 100% (60/60), done.
remote: Total 177 (delta 43), reused 68 (delta 34), pack-reused 80
Receiving objects: 100% (177/177), 87.68 KiB | 568.00 KiB/s, done.
Resolving deltas: 100% (63/63), done.

Navigate to the folder which contains the helm chart of tigris-stack:

cd tigris-deploy/helm/tigris-stack

Deploy Tigris Stack

To ensure there is initial quorum for Tigris Search, we should deploy it initially with a single replica.

❯ helm install tigris-stack . --set tigris-search.replicas=1
W1103 11:56:22.823655 12264 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
W1103 11:56:30.072806 12264 warnings.go:70] Autopilot increased resource requests for Deployment default/tigris-server to meet requirements. See http://g.co/gke/autopilot-resources.
W1103 11:56:30.089432 12264 warnings.go:70] Autopilot increased resource requests for Deployment default/tigris-stack-fdb-operator to meet requirements. See http://g.co/gke/autopilot-resources.
W1103 11:56:30.232424 12264 warnings.go:70] Autopilot set default resource requests on StatefulSet default/tigris-search for container tigris-ts-node-mgr, as resource requests were not specified, and adjusted resource requests to meet requirements. See http://g.co/gke/autopilot-defaults and http://g.co/gke/autopilot-resources.
NAME: tigris-stack
LAST DEPLOYED: Thu Nov 3 11:56:25 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

At this point your cluster will likely only have a few nodes:

❯ kubectl get nodes
W1103 11:57:04.068108 12352 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
NAME STATUS ROLES AGE VERSION
gk3-doc-default-pool-ddd321b8-4v8x Ready <none> 42h v1.23.8-gke.1900
gk3-doc-default-pool-e88cea62-9b77 Ready <none> 42h v1.23.8-gke.1900

The pods will be in the Pending state and trigger pod scale-ups:

❯ kubectl get pods
W1103 11:56:43.749022 12327 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
NAME READY STATUS RESTARTS AGE
tigris-search-0 0/2 Pending 0 14s
tigris-server-8646cb4b7b-fz6h4 0/1 Pending 0 14s
tigris-server-8646cb4b7b-hmxj9 0/1 Pending 0 14s
tigris-server-8646cb4b7b-qsjw7 0/1 Pending 0 14s
tigris-stack-fdb-operator-8fd845b9-wb4r5 0/1 Pending 0 14s


❯ kubectl describe pod tigris-search-0 | tail
W1103 11:58:18.395905 12695 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
Node-Selectors: <none>
Tolerations: kubernetes.io/arch=amd64:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 108s gke.io/optimize-utilization-scheduler 0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory.
Warning FailedScheduling 38s gke.io/optimize-utilization-scheduler 0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 2 Insufficient cpu, 2 Insufficient memory.
Normal TriggeredScaleUp 26s cluster-autoscaler pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/mystic-berm-360500/zones/us-west2-a/instanceGroups/gk3-doc-nap-10cyk06a-9f9e9a3f-grp 0->1 (max: 1000)}]

Tigris will restart a few times before it changes state to Running. This is due to the unavailability of FoundationDB, the key-value store Tigris uses for persistence.

As you can see below, fdb is still in a Pending state when the tigris-server Pods are already up:

❯ kubectl get pods
W1103 12:05:30.762386 14893 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
NAME READY STATUS RESTARTS AGE
fdb-cluster-log-1 0/2 Pending 0 43s
fdb-cluster-log-2 0/2 Pending 0 43s
fdb-cluster-log-3 0/2 Pending 0 42s
fdb-cluster-log-4 0/2 Pending 0 42s
fdb-cluster-log-5 0/2 Pending 0 42s
fdb-cluster-stateless-1 0/2 Pending 0 43s
fdb-cluster-stateless-10 0/2 Pending 0 43s
fdb-cluster-stateless-2 0/2 Pending 0 43s
fdb-cluster-stateless-3 0/2 Pending 0 43s
fdb-cluster-stateless-4 0/2 Pending 0 43s
fdb-cluster-stateless-5 0/2 Pending 0 43s
fdb-cluster-stateless-6 0/2 Pending 0 43s
fdb-cluster-stateless-7 0/2 Pending 0 43s
fdb-cluster-stateless-8 0/2 Pending 0 43s
fdb-cluster-stateless-9 0/2 Pending 0 43s
fdb-cluster-storage-1 0/2 Pending 0 43s
fdb-cluster-storage-2 0/2 Pending 0 43s
fdb-cluster-storage-3 0/2 Pending 0 43s
fdb-cluster-storage-4 0/2 Pending 0 43s
fdb-cluster-storage-5 0/2 Pending 0 43s
tigris-search-0 2/2 Running 1 (5m49s ago) 9m1s
tigris-server-8646cb4b7b-fz6h4 0/1 ContainerCreating 0 9m1s
tigris-server-8646cb4b7b-hmxj9 0/1 CrashLoopBackOff 1 (6s ago) 9m1s
tigris-server-8646cb4b7b-qsjw7 0/1 CrashLoopBackOff 2 (7s ago) 9m1s
tigris-stack-fdb-operator-8fd845b9-zgr4t 1/1 Running 0 5m55s

:info: You can improve the deployment sequence by using more sophisticated deployment methods, such as Synchronization Waves in ArgoCD!

Give Autopilot enough time to scale up nodes for the deployment. FoundationDB will likely trigger a separate scale-up event on its own.

❯ kubectl get nodes
W1103 12:09:59.375610 16639 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
NAME STATUS ROLES AGE VERSION
gk3-doc-default-pool-ddd321b8-4v8x Ready <none> 42h v1.23.8-gke.1900
gk3-doc-default-pool-e88cea62-9b77 Ready <none> 42h v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-854c84a8-4qss Ready <none> 4m23s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-854c84a8-6fd2 Ready <none> 4m21s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-854c84a8-m6hp Ready <none> 4m23s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-854c84a8-p8zq Ready <none> 4m21s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-854c84a8-r744 Ready <none> 4m22s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-854c84a8-xj5b Ready <none> 4m20s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-9f9e9a3f-4m2r Ready <none> 4m18s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-9f9e9a3f-d6nm Ready <none> 4m18s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-9f9e9a3f-ggxv Ready <none> 4m17s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-9f9e9a3f-lfwl Ready <none> 4m18s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-9f9e9a3f-s456 Ready <none> 4m18s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-9f9e9a3f-slg8 Ready <none> 4m19s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-9f9e9a3f-vg27 Ready <none> 11m v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-9f9e9a3f-xf4k Ready <none> 4m18s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-9f9e9a3f-xptm Ready <none> 4m18s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-c0284c87-5hpx Ready <none> 4m13s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-c0284c87-96c2 Ready <none> 4m12s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-c0284c87-c7h8 Ready <none> 4m13s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-c0284c87-klm4 Ready <none> 4m12s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-c0284c87-mrqp Ready <none> 4m12s v1.23.8-gke.1900
gk3-doc-nap-10cyk06a-c0284c87-wwj2 Ready <none> 4m12s v1.23.8-gke.1900
gk3-doc-nap-qm2jb0jm-1393ada1-bgwt Ready <none> 11m v1.23.8-gke.1900
gk3-doc-nap-qm2jb0jm-6d70fd3a-pxdr Ready <none> 12m v1.23.8-gke.1900

Following the scale up of the nodes, the services to slowly also come up. As it is waiting for foundational services to start,

However, after about 15 minutes the Pods should become available:

❯ kubectl get pods
W1103 12:10:45.077224 16929 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
NAME READY STATUS RESTARTS AGE
fdb-cluster-log-1 2/2 Running 0 5m57s
fdb-cluster-log-2 2/2 Running 0 5m57s
fdb-cluster-log-3 2/2 Running 0 5m56s
fdb-cluster-log-4 2/2 Running 0 5m56s
fdb-cluster-log-5 2/2 Running 0 5m56s
fdb-cluster-stateless-1 2/2 Running 0 5m57s
fdb-cluster-stateless-10 2/2 Running 0 5m57s
fdb-cluster-stateless-2 2/2 Running 0 5m57s
fdb-cluster-stateless-3 2/2 Running 0 5m57s
fdb-cluster-stateless-4 2/2 Running 0 5m57s
fdb-cluster-stateless-5 2/2 Running 0 5m57s
fdb-cluster-stateless-6 2/2 Running 0 5m57s
fdb-cluster-stateless-7 2/2 Running 0 5m57s
fdb-cluster-stateless-8 2/2 Running 0 5m57s
fdb-cluster-stateless-9 2/2 Running 0 5m57s
fdb-cluster-storage-1 2/2 Running 0 5m57s
fdb-cluster-storage-2 2/2 Running 0 5m57s
fdb-cluster-storage-3 2/2 Running 0 5m57s
fdb-cluster-storage-4 2/2 Running 0 5m57s
fdb-cluster-storage-5 2/2 Running 0 5m57s
tigris-search-0 2/2 Running 1 (11m ago) 14m
tigris-server-8646cb4b7b-95lcf 1/1 Running 0 2m37s
tigris-server-8646cb4b7b-gff64 1/1 Running 2 (3m12s ago) 3m23s
tigris-server-8646cb4b7b-hmxj9 1/1 Running 5 (3m59s ago) 14m
tigris-stack-fdb-operator-8fd845b9-zgr4t 1/1 Running 0 11m

That’s it, your Tigris deployment should be now on its way coming up!

Validate Deployment

This time we are going to validate Tigris Server using the Tigris CLI, using a small linux Pod that was deployed in the same namespace as the Tigris Stack.

First we need to install the CLI:

$ curl -sSL https://tigris.dev/cli-linux | sudo tar -xz -C /usr/local/bin
...
$ ls -la /usr/local/bin/tigris
-rwxr-xr-x 1 1001 121 17264640 Nov 3 07:21 /usr/local/bin/tigris

Set TIGRIS_URL to point at the Service endpoint of tigris-server:

$ export TIGRIS_URL=http://tigris-http:80

After that see if you can interact with the Tigris database using the tigris utility:

$ tigris quota limits
{
"ReadUnits": 100,
"WriteUnits": 25,
"StorageSize": 104857600
}

$ tigris server info
{
"server_version": "v1.0.0-beta.17"
}

$ tigris server version
tigris server version at http://tigris-http:80 is v1.0.0-beta.17

$ tigris create database robert

$ tigris list databases
robert

Preparing For Production

Scaling Search Out

To ensure Search is also redundant, once the deployment has progressed past transient state, Tigris Search should be scaled up to multiple replicas. In order to maintain quorum, the number of replicas should be set to an odd number, at a minimum of 3.

Below command will increase the number of Tigris Search replicas to 5 which should be a sufficiently large number of replicas for an initial Production deployment:

❯ helm upgrade tigris-stack . --set tigris-search.replicas=5
W1103 18:12:06.790278 82440 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
W1103 18:12:14.011524 82440 warnings.go:70] Autopilot increased resource requests for Deployment default/tigris-stack-fdb-operator to meet requirements. See http://g.co/gke/autopilot-resources.
W1103 18:12:14.362641 82440 warnings.go:70] Autopilot increased resource requests for Deployment default/tigris-server to meet requirements. See http://g.co/gke/autopilot-resources.
W1103 18:12:14.711610 82440 warnings.go:70] Autopilot increased resource requests for StatefulSet default/tigris-search to meet requirements. See http://g.co/gke/autopilot-resources.
Release "tigris-stack" has been upgraded. Happy Helming!
NAME: tigris-stack
LAST DEPLOYED: Thu Nov 3 18:12:08 2022
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

You can verify that additional replicas were started, using kubectl:

❯ kubectl get pods | grep tigris
W1103 18:12:33.301669 82537 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
tigris-search-0 2/2 Running 8 (25m ago) 6h16m
tigris-search-1 0/2 Pending 0 19s
tigris-search-2 0/2 Pending 0 19s
tigris-search-3 0/2 Pending 0 18s
tigris-search-4 0/2 Pending 0 18s
tigris-server-8646cb4b7b-95lcf 1/1 Running 0 6h4m
tigris-server-8646cb4b7b-gff64 1/1 Running 2 (6h5m ago) 6h5m
tigris-server-8646cb4b7b-hmxj9 1/1 Running 5 (6h5m ago) 6h16m
tigris-stack-fdb-operator-8fd845b9-zgr4t 1/1 Running 0 6h12m

The replicas should catch up quickly as there isn't a lot of search index to be synchronized. However, GKE Autopilot might need to scale up the nodes prior:

❯ kubectl describe pod tigris-search-1 | tail
W1103 18:14:04.069915 83269 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
Node-Selectors: <none>
Tolerations: kubernetes.io/arch=amd64:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 110s gke.io/optimize-utilization-scheduler 0/24 nodes are available: 24 Insufficient cpu, 24 Insufficient memory.
Normal TriggeredScaleUp 74s cluster-autoscaler pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/mystic-berm-360500/zones/us-west2-c/instanceGroups/gk3-doc-nap-2qbw2tfi-b7486e29-grp 0->1 (max: 1000)} {https://www.googleapis.com/compute/v1/projects/mystic-berm-360500/zones/us-west2-a/instanceGroups/gk3-doc-nap-2qbw2tfi-efcf60fb-grp 0->1 (max: 1000)}]
Warning FailedScheduling 23s gke.io/optimize-utilization-scheduler 0/26 nodes are available: 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 24 Insufficient cpu, 24 Insufficient memory.

It should take only a minute or two to get them up Running:

❯ kubectl get pods | grep tigris-search
W1103 18:15:05.957816 83699 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
tigris-search-0 2/2 Running 8 (27m ago) 6h18m
tigris-search-1 2/2 Running 0 2m52s
tigris-search-2 2/2 Running 0 2m52s
tigris-search-3 2/2 Running 0 2m51s
tigris-search-4 2/2 Running 0 2m51s

Ending TLS

For a Production installation you will want to add a certificate to your load balancer. However, as this step does not have any Tigris specific detail, we are going to skip detailing this step.

Wrapping Up!

I hope above could illustrate how easy it is to deploy Tigris to GKE Autopilot! Feel free to compare it to the article about deploying Tigris to EKS where we discussed the steps necessary to deploy it to AWS!

If you have any suggestions for us on Tigris related subjects that you think people might find interesting, feel free to reach out to us on either our Tigris Community Slack channel or our Tigris Discord server!

Hope you enjoyed reading or watching this blog or vlog! If you did, stay tuned as next we are going to cover a few interesting subjects such as performing logical backups and restores with Tigris!


Tigris is the data platform built for developers! Use it as a scalable, ACID transactional, real-time backend for your serverless applications. Build data-rich features without worrying about slow queries or missing indexes. Seamlessly implement search within your applications with its embedded search engine. Connect serverless functions with its event streams to build highly responsive applications that scale automatically.

Sign up for the beta

Get early access and try out Tigris for your next application. Join our Slack or Discord community to ask any questions you might have.

· 8 min read
Robert Barabas

This blog outlines the deployment of Tigris on an AWS managed Elastic Kubernetes Service (EKS). Future blogs we will walk through other aspects of the product setup for Tigris as a service, such as the setup of authentication. Stay tuned!

The installation will use recommended settings for redundancy, allocating more resources than a simple laptop based installation would. For more information on the laptop based installation please consult our previous blog!

If you would rather watch a video, check out the deployment in action on YouTube:

Requirements

Installation Host

The following components will need to be installed on the machine you are performing the deployment steps on:

  • Helm
  • AWS Cli

Helm will be used to install the Tigris Stack Chart:

❯ helm version
version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.19.2"}

AWS Cli will be used to setup a wildcard certificate:

❯ aws --version
aws-cli/2.8.5 Python/3.10.8 Darwin/21.6.0 source/arm64 prompt/off

EKS

Outside of the above you will need an EKS cluster with access and sufficient resources available for deployment.

❯ kubectl get nodes -L beta.kubernetes.io/instance-type
NAME STATUS ROLES AGE VERSION INSTANCE-TYPE
ip-10-2-10-204.us-west-2.compute.internal Ready <none> 18h v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-10-6.us-west-2.compute.internal Ready <none> 6d v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-11-166.us-west-2.compute.internal Ready <none> 18h v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-11-169.us-west-2.compute.internal Ready <none> 18h v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-12-79.us-west-2.compute.internal Ready <none> 6d v1.21.14-eks-ba74326 r6i.2xlarge
ip-10-2-13-192.us-west-2.compute.internal Ready <none> 6d v1.21.14-eks-ba74326 r6i.2xlarge

Most of the resources will be consumed by FoundationDB and TypeSense, both of which are main building blocks to Tigris.

The EKS cluster must have the following components installed:

  • AWS Load Balancer Controller
  • Cert Manager
  • EBS CSI Controller
❯ helm list -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
aws-load-balancer-controller kube-system 1 2022-10-20 11:00:30.800715 -0400 EDT deployed aws-load-balancer-controller-1.4.4 v2.4.3
cert-manager kube-system 1 2022-10-20 11:00:27.772092 -0400 EDT deployed cert-manager-v1.6.1 v1.6.1
cert-manager-ca kube-system 1 2022-10-20 11:01:45.36529 -0400 EDT deployed cert-manager-ca-0.2.0 v0.1.0
external-dns kube-system 1 2022-10-20 11:00:25.898907 -0400 EDT deployed external-dns-6.7.1 0.12.1
metrics-server kube-system 1 2022-10-20 11:00:26.973139 -0400 EDT deployed metrics-server-3.8.1 0.6.1

Deployment

The installation deploys the following components:

  • Kubernetes Operator for FoundationDB
  • FoundationDB
  • Tigris Search (TypeSense)
  • Tigris Server

You can install the components individually or together, using the encompassing tigris-stack Helm Chart. Below I’m going to use this Chart to install Tigris.

Create Certificate for TLS

First, we need to generate a certificate for TLS. This certificate will be used on the load balancer to terminate TLS connections:

❯ aws acm request-certificate --domain-name='*.example.com'
{
"CertificateArn": "arn:aws:acm:us-west-2:878843336588:certificate/fe257207-b117-4db0-ad6b-eef8d66308cd"
}

Prepare For Deployment

Next, check out the deploy script repository:

git clone git@github.com:tigrisdata/tigris-deploy.git
Cloning into 'tigris-deploy'...
remote: Enumerating objects: 177, done.
remote: Counting objects: 100% (97/97), done.
remote: Compressing objects: 100% (60/60), done.
remote: Total 177 (delta 43), reused 68 (delta 34), pack-reused 80
Receiving objects: 100% (177/177), 87.68 KiB | 568.00 KiB/s, done.
Resolving deltas: 100% (63/63), done.

Navigate to the folder which contains the helm chart of tigris-stack:

cd tigris-deploy/helm/tigris-stack

Deploy Tigris Stack

As we are deploying to EKS, we are going to use an ALB as our load balancer. The host for this installation will be set to api.example.com.

⚠️ You will want to make sure that above hostname matches the domain of the wildcard certificate you created previously!

Finally, to ensure there is initial quorum for Tigris Search, we deploy it initially with a single replica:

❯ helm install tigris-stack . -f ./values.yaml --set tigris-server.ingress_aws.enabled=true --set tigris-server.tls_hostname="api.example.com" --set tigris-search.replicas=1

NAME: tigris-stack
LAST DEPLOYED: Tue Oct 25 18:58:53 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

That’s it, your Tigris deployment should be now on its way coming up!

Validate Deployment

Generally speaking, there are two high level checkboxes to check. First, we should ensure that all the resources were scheduled and are available and running. Second, we will want to make sure that Tigris API is accessible using the Ingress resource allocated. These steps are expanded upon below.

Resources Validation

Allow the resources such as FoundationDB a couple minutes to initialize. In a Production-ready installation FoundationDB would allocate significant resources. You will want to make sure that the FDB Pods were able to be scheduled.

Tigris will not enter the Running state until FoundationDB becomes fully functional. It might take a couple minutes for FoundationDB to become unavailable, even when all of its Pods appear to be in the Running state.

You can validate if the FDB key-value store is available using fdbcli:

tigris@tigris-server-58ccd7bb9f-czcbb:/server$ fdbcli -C /mnt/fdb-config-volume/cluster-file
Using cluster file `/mnt/fdb-config-volume/cluster-file'.

The database is available.

Welcome to the fdbcli. For help, type `help'.
fdb>

Look for the message “The database is available” in fdbcli’s output.

Final Overview

Unless there were additional customizations, your output should be similar to below:

❯ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/fdb-cluster-log-1 2/2 Running 0 5m53s
pod/fdb-cluster-log-2 2/2 Running 0 5m53s
pod/fdb-cluster-log-3 2/2 Running 0 5m53s
pod/fdb-cluster-log-4 2/2 Running 0 5m53s
pod/fdb-cluster-log-5 2/2 Running 0 5m53s
pod/fdb-cluster-stateless-1 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-10 2/2 Running 0 5m53s
pod/fdb-cluster-stateless-2 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-3 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-4 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-5 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-6 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-7 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-8 2/2 Running 0 5m54s
pod/fdb-cluster-stateless-9 2/2 Running 0 5m53s
pod/fdb-cluster-storage-1 2/2 Running 0 5m54s
pod/fdb-cluster-storage-2 2/2 Running 0 5m54s
pod/fdb-cluster-storage-3 2/2 Running 0 5m54s
pod/fdb-cluster-storage-4 2/2 Running 0 5m54s
pod/fdb-cluster-storage-5 2/2 Running 0 5m54s
pod/tigris-search-0 2/2 Running 1 6m49s
pod/tigris-server-58ccd7bb9f-czcbb 1/1 Running 4 6m49s
pod/tigris-server-58ccd7bb9f-ngjk5 1/1 Running 4 6m49s
pod/tigris-server-58ccd7bb9f-rnbxb 1/1 Running 4 6m49s
pod/tigris-stack-fdb-operator-5d9dbc4c9d-ptlng 1/1 Running 0 6m49s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 5d8h
service/tigris-grpc NodePort 172.20.60.127 <none> 80:30440/TCP 6m50s
service/tigris-headless ClusterIP None <none> 8080/TCP 6m50s
service/tigris-http NodePort 172.20.82.191 <none> 80:30675/TCP 6m50s
service/tigris-search NodePort 172.20.130.194 <none> 80:31720/TCP 6m50s
service/ts ClusterIP None <none> 8108/TCP 6m50s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tigris-server 3/3 3 3 6m50s
deployment.apps/tigris-stack-fdb-operator 1/1 1 1 6m50s

NAME DESIRED CURRENT READY AGE
replicaset.apps/tigris-server-58ccd7bb9f 3 3 3 6m50s
replicaset.apps/tigris-stack-fdb-operator-5d9dbc4c9d 1 1 1 6m50s

NAME READY AGE
statefulset.apps/tigris-search 1/1 6m50s

Ingress Validation

The most EKS installation specific piece to a Tigris installation is generally the load balancer and related resources.

The installation will create an annotated Ingress resource:

❯ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
tigris-server <none> * 80 3m25s
❯ kubectl get ingress tigris-server -o yaml | grep -A17 annotations:
annotations:
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/conditions.tigris-grpc: |
[{"field":"http-header","httpHeaderConfig":{"httpHeaderName": "Content-Type", "values":["application/grpc"]}}]
alb.ingress.kubernetes.io/group.name: tigris-server-lb
alb.ingress.kubernetes.io/healthcheck-path: /v1/health
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/load-balancer-attributes: routing.http.drop_invalid_header_fields.enabled=true
alb.ingress.kubernetes.io/load-balancer-name: tigris-server-lb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
alb.ingress.kubernetes.io/tags: service=tigris-server
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.app_cookie.duration_seconds=10,stickiness.type=app_cookie,stickiness.app_cookie.cookie_name=Tigris-Tx-Id
alb.ingress.kubernetes.io/target-type: ip
external-dns.alpha.kubernetes.io/ingress-hostname-source: annotation-only
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: tigris-stack
meta.helm.sh/release-namespace: default

AWS Load Balancer Controller will create an ALB based on above annotation:

❯ aws elbv2 describe-load-balancers --names tigris-server-lb | grep -i dnsname
"DNSName": "tigris-server-lb-<redacted>.us-west-2.elb.amazonaws.com",

Make sure that your load balancer is healthy and operational before you proceed!

Preparing For Production

There is one last step that is required for a proper, production-ready installation.

In order to ensure there is proper redundancy under Tigris Search, you will want to increase the number of replicas to an odd number of replicas post initial installation.

An odd number of replicas is required to ensure that quorum can be reached. An even number of replicas could end up being without a tie breaker during network partitioning, where both partitions may end up with the same number of replicas.

Below command will increase the number of Tigris Search replicas to 5:

❯ helm upgrade tigris-stack . -f ./values.yaml --set tigris-server.ingress_aws.enabled=true --set tigris-server.tls_hostname="api.example.com" --set tigris-search.replicas=5
Release "tigris-stack" has been upgraded. Happy Helming!
NAME: tigris-stack
LAST DEPLOYED: Tue Oct 25 19:13:33 2022
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

We recommend using at least 3 replicas for Production installations with 5 being recommended for load handling purposes. This is the default setting in the tigris-stack Chart that is only reduced during install, to ensure quorum can be achieved during initialization.

You can verify that there are now 5 replicas running with a simple kubectl command:

❯ kubectl get pods | grep tigris
tigris-search-0 2/2 Running 1 17m
tigris-search-1 2/2 Running 0 2m39s
tigris-search-2 2/2 Running 0 2m39s
tigris-search-3 2/2 Running 0 2m39s
tigris-search-4 2/2 Running 0 2m39s
tigris-server-58ccd7bb9f-czcbb 1/1 Running 4 17m
tigris-server-58ccd7bb9f-ngjk5 1/1 Running 4 17m
tigris-server-58ccd7bb9f-rnbxb 1/1 Running 4 17m
tigris-stack-fdb-operator-5d9dbc4c9d-ptlng 1/1 Running 0 17m

Final Thoughts

In this blog we have covered the deployment of Tigris to one of the most common Managed Kubernetes platforms available in the Cloud. As you can see from above the deployment process is fairly easy and straightforward on EKS.

If you liked our blog make sure to follow us as next week we are going to cover the deployment of Tigris on Google Cloud Platform!

Enjoy using Tigris as much as we enjoy building it!!!


Tigris is the data platform built for developers! Use it as a scalable, ACID transactional, real-time backend for your serverless applications. Build data-rich features without worrying about slow queries or missing indexes. Seamlessly implement search within your applications with its embedded search engine. Connect serverless functions with its event streams to build highly responsive applications that scale automatically.

Sign up for the beta

Get early access and try out Tigris for your next application. Join our Slack or Discord community to ask any questions you might have.

· 8 min read
Peter Boros
Robert Barabas

Tigris is an open source developer data platform that makes building data-rich applications a breeze. This is the first of a series of blog posts where we show you how to deploy Tigris in various environments. In the first post of the series we will show you how to set up the Tigris platform on your laptop. In our next posts we will cover deploying Tigris on EKS and GKE.

We will use tigris-deploy, which is a set of helm charts and a wrapper script. Using it, we will deploy the complete Tigris stack including FoundationDB, TypeSense, Grafana, VictoriaMetrics, Tigris itself and an nginx-based load balancer. For this example, we will use minikube but you may use k3s or Kind, if you prefer.

If you don't want to use the provided metrics store and grafana, you can turn those off easily via values yaml. Tigris itself is providing prometheus compatible metrics on the /metrics url.

Watch the video below to see the deployment in action

Clone tigris-deploy repository

The quickest way to deploy Tigris is using its Helm Chart. To perform a Helm-based installation, you need to clone the tigris-deploy repository. This has several helm charts in it. One of them is tigris-stack, a “chart of charts” that installs all the dependencies, so installing that will result in a fully working Tigris platform.

git clone git@github.com:tigrisdata/tigris-deploy.git
cd tigris-deploy

Start minikube

In this example, we will use minikube to have a kubernetes cluster. The goal of this step is to have a working kubernetes cluster and a kubernetes client that is configured to use that cluster.

minikube start --kubernetes-version=1.21.14
Output
😄  minikube v1.27.1 on Darwin 12.6
✨ Automatically selected the docker driver
📌 Using Docker Desktop driver with root privileges
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=16300MB) ...
🐳 Preparing Kubernetes v1.21.14 on Docker 20.10.18 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

After minikube is started, no pods will run in the default namespace.

kubectl get pods
Output
No resources found in default namespace.

But you should see pods in the kube-system namespace.

kubectl get pods -A
Output
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system coredns-558bd4d5db-7x8hv 1/1 Running 0 82s
kube-system etcd-minikube 1/1 Running 0 94s
kube-system kube-apiserver-minikube 1/1 Running 0 94s
kube-system kube-controller-manager-minikube 1/1 Running 0 94s
kube-system kube-proxy-69xh5 1/1 Running 0 82s
kube-system kube-scheduler-minikube 1/1 Running 0 94s
kube-system storage-provisioner 1/1 Running 0 92s

At this point, we have minikube working, ready to deploy tigris in it.

Deploy Tigris

For the sake of simplicity, we provide deploy.sh, a wrapper script. This script extends the Chart's functionality and addresses common tasks such as the setup of dependencies for the Chart.

We will deploy Tigris with no redundancy to minimize resource consumption and make Tigris fit on a reasonably equipped laptop.

For Production, you will want to enable full redundancy. Without overrides, the Charts will deploy resources with redundancy enabled.

bash deploy.sh
Output
Getting updates for unmanaged Helm repositories...
...Successfully got an update from the "https://kubernetes.github.io/ingress-nginx" chart repository
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "vm" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 7 charts
Downloading ingress-nginx from repo https://kubernetes.github.io/ingress-nginx
Downloading victoria-metrics-single from repo https://victoriametrics.github.io/helm-charts
Downloading grafana from repo https://grafana.github.io/helm-charts
Deleting outdated charts
Release "tigris-stack" does not exist. Installing it now.
W1014 17:51:00.854310 48946 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 17:51:00.856463 48946 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 17:51:00.858829 48946 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 17:51:09.081654 48946 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 17:51:09.081939 48946 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1014 17:51:09.082027 48946 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: tigris-stack
LAST DEPLOYED: Fri Oct 14 17:50:59 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1

The platform should become available 4-5 minutes after the deploy script is run. This can be verified with kubectl get all. Once It’s ready, you will see this output.

kubectl get all
Output
NAME                                                         READY   STATUS    RESTARTS   AGE
pod/fdb-cluster-log-1 2/2 Running 0 2m18s
pod/fdb-cluster-stateless-1 2/2 Running 0 2m18s
pod/fdb-cluster-storage-1 2/2 Running 0 2m18s
pod/tigris-search-0 2/2 Running 1 4m45s
pod/tigris-server-79f77c8cb7-cxwxk 1/1 Running 0 4m45s
pod/tigris-stack-fdb-operator-5d9dbc4c9d-6f6b2 1/1 Running 0 4m45s
pod/tigris-stack-grafana-7586c54dc-24fvn 1/1 Running 0 4m45s
pod/tigris-stack-ingress-nginx-controller-57c4689667-th296 1/1 Running 0 4m45s
pod/tigris-stack-victoria-metrics-single-server-0 1/1 Running 0 4m45s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9m7s
service/tigris-grpc NodePort 10.109.166.16 <none> 80:31740/TCP 4m45s
service/tigris-headless ClusterIP None <none> 8080/TCP 4m45s
service/tigris-http NodePort 10.100.38.18 <none> 80:31945/TCP 4m45s
service/tigris-search NodePort 10.102.197.16 <none> 80:32466/TCP 4m45s
service/tigris-stack-grafana ClusterIP 10.111.119.96 <none> 80/TCP 4m45s
service/tigris-stack-ingress-nginx-controller LoadBalancer 10.102.175.141 <pending> 80:31965/TCP,443:31050/TCP 4m45s
service/tigris-stack-ingress-nginx-controller-admission ClusterIP 10.100.243.2 <none> 443/TCP 4m45s
service/tigris-stack-victoria-metrics-single-server ClusterIP None <none> 8428/TCP 4m45s
service/ts ClusterIP None <none> 8108/TCP 4m45s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tigris-server 1/1 1 1 4m45s
deployment.apps/tigris-stack-fdb-operator 1/1 1 1 4m45s
deployment.apps/tigris-stack-grafana 1/1 1 1 4m45s
deployment.apps/tigris-stack-ingress-nginx-controller 1/1 1 1 4m45s

NAME DESIRED CURRENT READY AGE
replicaset.apps/tigris-server-79f77c8cb7 1 1 1 4m45s
replicaset.apps/tigris-stack-fdb-operator-5d9dbc4c9d 1 1 1 4m45s
replicaset.apps/tigris-stack-grafana-7586c54dc 1 1 1 4m45s
replicaset.apps/tigris-stack-ingress-nginx-controller-57c4689667 1 1 1 4m45s

NAME READY AGE
statefulset.apps/tigris-search 1/1 4m45s
statefulset.apps/tigris-stack-victoria-metrics-single-server 1/1 4m45s

Here is an example output when the service is not ready yet.

Output
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/tigris-search-0 0/2 ContainerCreating 0 40s
pod/tigris-server-79f77c8cb7-cxwxk 0/1 ContainerCreating 0 40s
pod/tigris-stack-fdb-operator-5d9dbc4c9d-6f6b2 0/1 Init:0/3 0 40s
pod/tigris-stack-grafana-7586c54dc-24fvn 0/1 Init:0/1 0 40s
pod/tigris-stack-ingress-nginx-controller-57c4689667-th296 1/1 Running 0 40s
pod/tigris-stack-victoria-metrics-single-server-0 0/1 Running 0 40s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5m2s
service/tigris-grpc NodePort 10.109.166.16 <none> 80:31740/TCP 40s
service/tigris-headless ClusterIP None <none> 8080/TCP 40s
service/tigris-http NodePort 10.100.38.18 <none> 80:31945/TCP 40s
service/tigris-search NodePort 10.102.197.16 <none> 80:32466/TCP 40s
service/tigris-stack-grafana ClusterIP 10.111.119.96 <none> 80/TCP 40s
service/tigris-stack-ingress-nginx-controller LoadBalancer 10.102.175.141 <pending> 80:31965/TCP,443:31050/TCP 40s
service/tigris-stack-ingress-nginx-controller-admission ClusterIP 10.100.243.2 <none> 443/TCP 40s
service/tigris-stack-victoria-metrics-single-server ClusterIP None <none> 8428/TCP 40s
service/ts ClusterIP None <none> 8108/TCP 40s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tigris-server 0/1 1 0 40s
deployment.apps/tigris-stack-fdb-operator 0/1 1 0 40s
deployment.apps/tigris-stack-grafana 0/1 1 0 40s
deployment.apps/tigris-stack-ingress-nginx-controller 1/1 1 1 40s

NAME DESIRED CURRENT READY AGE
replicaset.apps/tigris-server-79f77c8cb7 1 1 0 40s
replicaset.apps/tigris-stack-fdb-operator-5d9dbc4c9d 1 1 0 40s
replicaset.apps/tigris-stack-grafana-7586c54dc 1 1 0 40s
replicaset.apps/tigris-stack-ingress-nginx-controller-57c4689667 1 1 1 40s

NAME READY AGE
statefulset.apps/tigris-search 0/1 40s
statefulset.apps/tigris-stack-victoria-metrics-single-server 0/1 40s

From the services part of the kubectl get all output, it’s visible that the tigris-http and the tigris-grpc services are available in kubernetes. Tigris provides the same API with HTTP and GRPC. We will use the GRPC with the tigris command line client to try it out.

Testing Tigris

We will use minikube to make these services available on our computer.

minikube service tigris-grpc --url
Output
http://127.0.0.1:52999
❗ Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

This exposes tigris service on port 52999. We can use the tigris command line client to interact with the platform running in kubernetes. Apart from this, now everybody knows I was writing this blog post using macOS :)

export TIGRIS_URL="127.0.0.1:52999"
tigris list databases
tigris create database foobar

Now when we should be able to see the database we just created

tigris list databases
Output
foobar

With our small interaction there, we listed the available databases twice and created one. Now let’s check what metrics we have about these interactions. To do that, we need to interact with the grafana service that was deployed by tigris-deploy. First, we need to fetch grafana’s admin password.

kubectl get secret tigris-stack-grafana -o yaml | awk '/admin-password/ {print $2}' | base64 --decode ; echo

The actual output of this command will be the password that can be used to access grafana. Now we can access the grafana service with minikube service just like before.

minikube service tigris-stack-grafana --url
Output
😿  service default/tigris-stack-grafana has no node port
http://127.0.0.1:53122
❗ Because you are using a Docker driver on darwin, the terminal needs to be open to run it.

After logging in with the admin password, click on explore to access the metrics.

Grafana explore button

The VictoriaMetrics instance stores the metrics and is pre-configured as a data source. Switch the query input field to code from the builder.

Grafana explore screen

Once this is done, check the requests_count_ok metrics.

Grafana explore metric

By checking requests_count_ok, we can view the metrics about the 3 requests we just issued. For further information, check out the documentation on metrics, as well as the documentation on deployment.


Tigris is the data platform built for developers! Use it as a scalable, ACID transactional, real-time backend for your serverless applications. Build data-rich features without worrying about slow queries or missing indexes. Seamlessly implement search within your applications with its embedded search engine. Connect serverless functions with its event streams to build highly responsive applications that scale automatically.

Sign up for the beta

Get early access and try out Tigris for your next application. Join our Slack or Discord community to ask any questions you might have.