Kubernetes Deployment

The Components

The method Keyless adopts to ship its Infrastructure Components is via Helm Chart deployment.

The following diagram illustrates a general overview of the Infrastructure Components

The Keyless Infrastructure is composed of the following components:

Keyless Node: responsible for implementing the distributed privacy-preserving protocol for Keyless zero-proof biometric authentication

Node Persistence Service: service implementing a database abstraction layer (Postgres and OracleDB currently supported)

Database: database used by Keyless to store user data

Circuit Storage Service: service offering internal-use REST APIs to allow Keyless node to store circuits needed to authenticate users

Database + S3 Storage: used by Circuit Storage Service to store circuits and metadata

Keyless Rest API (Operations API): offers REST APIs for transaction confirmation and other backend services.

Each of the Keyless components has it's own chart and can be deployed standalone, but we strongly suggest to use our Main Keylessd Helm Chart to deploy all needed components as one, more on this in the Deployment Steps section

Access the Repositories

The customer is going to use 2 repositories:

A Docker Image Registry: quay.io/keyless.technologies

A Helm Chart Registry: keylesstech.github.io

To access the Docker repository the following command needs to be executed:

$ docker login quay.io -u < username > -p < password >

There is no need for authentication against the Helm Chart Registry.

Deployment Steps

The first step is to add our helm repository to your helm repository list with the following command

$ helm repo add keyless https://keylesstech.github.io

Then you can list the charts

$ helm search repo keyless

And get the values.yaml from the keylessd chart, which is our global one.

$ helm show values keyless/keylessd

Our suggestion, to perform a deployment, is to include our main helm chart, keylessd as a dependency of your own chart so that you can decide how to provide secret creation and other dependencies that you may have. ( for example you may decide to deploy custom monitoring agents alongside keylessd).

Once you’re satisfied with your configuration you can simply apply it with

$ helm upgrade --install <release-name (for example keylessd)> keyless/keylessd

Configuration

The Deployment configuration is customizable using the helm values.yaml file which is divided into the following sections: 

global: {}

keylessd:

  circuit-storage:
    ...
  node:
    ...
  operations-api:
    ...
  node-persistence:
    ...

global : global values such as :

  • namespace: namespace name where to deploy the resources ( optional ), defaults to default

  • token: to authenticate to the registry ( optional )

  • createNamespace: to create the namespace if it doesn't already exists

circuit-storage: configuration specific to Circuit Storage Service

node: configuration specific to Keyless Node

operations-api: configuration for the Operation API component

node-persistence: configuration for the Node Persistence API component

Global

As stated above, the global section is optional and it is used if you do not desire to deploy each component into its own namespace but prefer to have all in a single namespace. You will then need to provide the name of the namespace and to set createNamespace variable to true as shown below

global:
  token: # eyJhd.....................
  namespace: keyless
  createNamespace: true

global.token variable is used if you want our chart to take care of creating the imagePullSecret secret, used by the Deployment to access the docker registry, usually when the intention is to deploy each service into its separate namespace.

Infact as you may know, to be able to reference a secret used to pull docker images, that secret needs to be created inside each namespace.

Otherwise you will be tasked with creating the secret registry of type kubernetes.io/dockerconfigjson so that it can be accessed by the Deployment.

To know more about the format of the secret registry please refere to the following link

https://kubernetes.io/docs/concepts/configuration/secret/#docker-config-secrets

If your registry doesn't require authentication, ( for example if syncronization between our Docker Registry and yours is already set up ) then the token variable is not needed.

Circuit Storage Service

As this service is tasked with reading/writing from an s3-compatible storage it will require access key and secret key to be available. It will also require a database to hold metadata. If you want to supply the corresponding Secret to hold the credentials, skip this section.

Expose Circuit Storage Service

keylessd:
  circuit-storage:
    publicRoute: true
    createRoute: true
    createIngress: false
    host: circuit-storage
    domain: keyless.local

To expose Circuit Storage service outside the Cluster publicRoute variable may be used.

In addition to this, it is possible to specify if Kubernetes should create an Ingress or a Route , depending if you're running on a vanilla cluster or on Openshift.

As Circuit Storage service is only a backend service, the default values are all false.

In case the values are set to true you will also need to set the following two variables:

  • host

  • domain

So that the corresponding Ingress/Route Object will be configured to get traffic from <host>.<domain>.

Custom Certificate to Access the Database or the S3-Compatible Backend

keylessd:
  circuit-storage:
    customCA: true
    certificates:
      - name: storage-cert
        mountPath: "/etc/ssl/custom-ca/storage-cert.pem"
        subPath: "storage-cert.pem"
        readOnly: true
      - name: db-cert
        mountPath: "/etc/ssl/custom-ca/db-cert.pem"
        subPath: "db-cert.pem"
        readOnly: true

We support custom CA certificates in case it’s needed, setting the customCA variable to true and adding the following ConfigMap object in your chart

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: storage-cert
  namespace: my-namespace-name
data:
  storage-cert.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: db-cert
  namespace: my-namespace-name
data:
  db-cert.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

This could be used in case the s3-compatible backend exposes a secure connection with a certificate or if the database requires additional authentication requirements.

The certificates will be mounted to a specific location in the pod and added at runtime in the internal keystore of our application.

The mountPath must always be in the format /etc/ssl/custom-ca/<certificate>.pem

Exposed Java SpringBoot Endpoints

The endpoints section of the values.yaml file, allows you to customize the amount of endpoints exposed by Circuit Storage, usually the defaults are set to expose env variables ( passwords are not shown), metrics and health-check.

keylessd:
  circuit-storage:
    configMap:
      endpoints:
        web: health,env,metrics
        metrics: true
        health: true
        env: true
        ...

Datadog Integration

We support Datadog Integration for Circuit Storage by setting up the datadogApiKey in the secrets section and the datadog.uri ( which may vary depending on whether you use datadog.eu, datadog.com etc.)

keylessd:
  circuit-storage:
    configMap:
      datadog:
        uri: https://api.datadoghq.eu

    secrets:
      datadogApiKey: "..."

In case you do not wish to use this integration, leave blank

Adding Java Options to Circuit Storage

The javaOpts variable allows the customization of java options which may need to be adjusted according to particular needs.

keylessd:
  circuit-storage:
    configMap:
      javaOpts: >-
        -XX:MaxRAMPercentage=75.0
        -Dspring.profiles.active=oracle,onprem
        -Dlogging.level.com.amazonaws=DEBUG

Database Connection

configMap.springDatasource.url variable holds the jdbc-formatted string to allow Circuit Storage to connect to its own database. The Instance needs a dedicated schema to be available ( default “circuit-storage”)

keylessd:
  circuit-storage:
    configMap:
      springDatasource:
        # Oracle
        url: jdbc:oracle:thin:@//<hostname>:1521/<service>
        # or Postgres
        url: jdbc:postgresql://<hostname>:5432/<dbName>

Database Initialization

To manage the database schema Circuit Storage makes use of the Flyway library.

keylessd:
  circuit-storage:
    configMap:
      springFlyway:
        enabled: true
        locations: oracle

configMap.springFlyway.enabled variable allows you to enable or disable database initialization scripts to be run at start time.

configMap.springFlyway.locations variable allows you to set “oracle” or “postgres” location depending on which database you’re using ( currently only the two mentioned are supported).

Disabling Flyway means that you'll need to initalize the database yourself, so please, if that's the case, reach out to us so that we can provide you the database initialization script to you

Mutual TLS Support for Oracle Database

Starting with release 0.1.4 it is possible to connect to Oracle Database using mutual TLS authentication.

To achive this the Oracle Wallet archive is needed.

To configure Circuit Storage to use Oracle Wallet follow this procedure:

  • Unzip the Oracle Wallet

unzip wallet.zip -d mywallet
Archive:  wallet.zip
  inflating: mywallet/ewallet.pem    
  inflating: mywallet/README         
  inflating: mywallet/cwallet.sso    
  inflating: mywallet/tnsnames.ora   
  inflating: mywallet/truststore.jks  
  inflating: mywallet/ojdbc.properties  
  inflating: mywallet/sqlnet.ora     
  inflating: mywallet/ewallet.p12    
  inflating: mywallet/keystore.jks

  • Execute the following command to create a Secret containing all files inside the wallet.

    In the example below the name of the secret will be "wallet"

$ kubectl create secret generic wallet --from-file=./mywallet/ -n <namespace-name>
secret/wallet created

  • Modify values.yaml, given the following tnsnames.ora example file

tnsnames.ora

circuitstoragetest_high = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
circuitstoragetest_low = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
circuitstoragetest_medium = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
circuitstoragetest_tp = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
circuitstoragetest_tpurgent = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))

and TNS_ADMIN=/network/admin as the directory which the wallet secret will be mounted into, as such:

keylessd:
  circuit-storage:
    mtls:
      secretName: wallet
    configMap:
      springDatasource:
        url: jdbc:oracle:thin:@circuitstoragetest_high?TNS_ADMIN=/network/admin

Remember that you will still need to provide database username and password as the following secrets:

  • secrets.datasourceUsername

  • secrets.datasourcePassword

Override Circuit Storage Resource Name

It is possible to override the default resource name of Circuit Storage using the fullnameOverride variable. Otherwise the resource name assumed will be “circuit-storage-<release-name>”.

Keyless Node

Expose Keyless Node Service

keylessd:
  node:
    publicRoute: true
    createRoute: false
    createIngress: true
    host: node
    domain: keyless.technology

To expose Keyless Node service outside the Cluster, publicRoute variable needs be used.

In addition to this, it is possible to specify if Kubernetes should create an Ingress or a Route , depending if you're running on a vanilla cluster or on Openshift.

As Keylessd Node needs to be exposed the default values for publicRoute, createRoute and createIngress are as shown above.

You will also need to set the following two variables:

  • host

  • domain

as shown above, so that the corresponding Ingress/Route Object will be configured to get traffic from <host>.<domain>.

As there can be edge cases where you would prefer to manage Ingresses outside our Chart it is possible to set publicRoute to false

In this case you'll need to provide the Ingress/Route configuration, but keep in mind that the traffic needs to flow from the Ingress controller to the Keyless Node in passthrough mode as the TLS termination happends at the node level.

As the Keyless node will use a certificate that will be mounted at runtime into the pod, to terminate the TLS connection, regardless of the value of publicRoute, it is required to specify the host and domain to be used, which need to corresponds to ones used to generate the certificate.

As an example, let’s say that our final endpoint is node.keyless.technology, and that we have a certificate generated using the correct CN (again node.keyless.technology) then in this case the host variable will correspond to "node", meanwhile domain will correspond to “keyless.technology”.

keylessd:
  node:
    host: node
    domain: keyless.technology

certName variable is used to reference the secret name which hosts the certificate.

keylessd:
  node:
    certName: node

The expected format for the certificate is the one refecenced here: https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets

configMap.nodeConfig.storageUrl variable is used to reference Circuit Storage service in the format http://<service-name>.<namespace>.svc.cluster.local

configMap.nodeConfig.databaseUrl variable is used to reference Node Persitence service in the format http://<service-name>.<namespace>.svc.cluster.local

keylessd:
  node:
    configMap:
      nodeConfig:
        storageUrl: http://circuit-storage-keylessd.keyless.svc.cluster.local
        databaseUrl: http://node-persistence-keylessd.keyless.svc.cluster.local

Override Keyless Node Resource Name

It is possible to override the default resource name of Keyless Node using the fullnameOverride variable. Otherwise the resource name assumed will be “node-<release-name>”.

Operations API

Expose Operations API Service

keylessd:
  operations-api:
    publicRoute: true
    createRoute: false
    createIngress: true
    host: operations-api
    domain: keyless.local

To expose Operations API service outside the Cluster, publicRoute variable can be used.

In addition to this, it is possible to specify if Kubernetes should create an Ingress or a Route , depending if you're running on a vanilla cluster or on Openshift.

As Operations API needs to be exposed the default values for publicRoute, createRoute and createIngress are as shown above.

You will also need to set the following two variables:

  • host

  • domain

as shown above, so that the corresponding Ingress/Route Object will be configured to get traffic from <host>.<domain>.

As there can be edge cases where you would prefer to manage Ingresses outside our Chart it is possible to set publicRoute to false

In this case you'll need to provide the Ingress/Route configuration, but keep in mind that the traffic needs to flow from the Ingress controller to Operations API in edge mode as the TLS termination happends at the cluster level.

configMap.nodePersistenceApiUrl variables a is used to reference NodePersitence service in the format http://<service-name>.<namespace>.svc.cluster.local

keylessd:
  operations-api:
    configMap:
      nodePersistenceApiUrl: https://node-persistence.default.svc.cluster.local:8080

Override Operations API Resource Name

It is possible to override the default resource name of Operations API using the fullnameOverride variable. Otherwise the resource name assumed will be “operations-api-<release-name>”.

Node Persistence

This service is tasked with reading/writing from/to a database. It will require a secret to be able access it.

Exposed Java SpringBoot Endpoints

The endpoints section of the values.yaml file, allows you to customize the amount of endpoints exposed by Node Persistence, usually the defaults are set to expose env variables ( passwords are not shown), metrics and health-check.

keylessd:
  node-persistence:
    configMap:
      endpoints:
        web: health,env,metrics
        metrics: true
        health: true
        env: true
        ...

Datadog Integration

We support Datadog Integration for Node Persistence by setting up the datadogApiKey in the secrets section and the datadog.uri ( which may vary depending on whether you use datadog.eu, datadog.com etc.)

keylessd:
  node-persistence:
    configMap:
      datadog:
        uri: https://api.datadoghq.eu

    secrets:
      datadogApiKey: "..."

In case you do not wish to use this integration, leave blank

Adding Java Options to Node Persistence

The javaOpts variable allows the customization of java options which may need to be adjusted according to particular needs.

keylessd:
  node-persistence:
    configMap:
      javaOpts: >-
        -XX:MaxRAMPercentage=75.0
        -Dlogging.level.com.amazonaws=DEBUG

Setting The Database Type

At the moment of writing Node Persistence supports Oracle and Postgres databases

To set which database type Node Persistence will need to use, it is possible to set springProfile variable to either "oracle" or "postgres"

keylessd:
  node-persistence:
    configMap:
      springProfile: oracle

Database Connection

configMap.springDatasource.url variable holds the jdbc-formatted string to allow Node Persistence to connect to its own database. The Instance needs a dedicated schema to be available ( default “keylessd”)

keylessd:
  node-persistence:
    configMap:
      springDatasource:
        # Oracle
        url: jdbc:oracle:thin:@//<hostname>:1521/<service>
        # or Postgres
        url: jdbc:postgresql://<hsotname>:5432/<dbName>

Database Initialization

To manage the database schema Node Persistence makes use of the Flyway library.

keylessd:
  node-persistence:
    configMap:
      springFlyway:
        enabled: true
        locations: oracle

configMap.springFlyway.enabled variable allows you to enable or disable database initialization scripts to be run at start time.

configMap.springFlyway.locations variable allows you to set “oracle” or “postgres” location depending on which database you’re using ( currently only the two mentioned are supported).

Disabling Flyway means that you'll need to initalize the database yourself, so please, if that's the case, reach out to us so that we can provide you the database initialization script to you

Mutual TLS Support for Oracle Database

Starting with release 0.1.4 it is possible to connect to Oracle Database using mutual TLS authentication.

To achive this the Oracle Wallet archive is needed.

To configure Node Persitence to use Oracle Wallet follow this procedure:

  • Unzip the Oracle Wallet

unzip wallet.zip -d mywallet
Archive:  wallet.zip
  inflating: mywallet/ewallet.pem    
  inflating: mywallet/README         
  inflating: mywallet/cwallet.sso    
  inflating: mywallet/tnsnames.ora   
  inflating: mywallet/truststore.jks  
  inflating: mywallet/ojdbc.properties  
  inflating: mywallet/sqlnet.ora     
  inflating: mywallet/ewallet.p12    
  inflating: mywallet/keystore.jks

  • Execute the following command to create a Secret containing all files inside the wallet.

    In the example below the name of the secret will be "wallet"

$ kubectl create secret generic wallet --from-file=./mywallet/ -n <namespace-name>
secret/wallet created

  • Modify values.yaml, given the following tnsnames.ora example file

tnsnames.ora

nodepersistencetest_high = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
nodepersistencetest_low = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
nodepersistencetest_medium = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
nodepersistencetest_tp = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
nodepersistencetest_tpurgent = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))

and TNS_ADMIN=/network/admin as the directory which the wallet secret will be mounted into, as such:

keylessd:
  node-persistence:
    mtls:
      secretName: wallet
    configMap:
      springDatasource:
        url: jdbc:oracle:thin:@nodepersistencetest_high?TNS_ADMIN=/network/admin

Remember that you will still need to provide database username and password as the following secrets:

  • secrets.datasourceUsername

  • secrets.datasourcePassword

Override Node Persistence Resource Name

It is possible to override the default resource name of Node Persistence using the fullnameOverride variable. Otherwise the resource name assumed will be “node-persistence-<release-name>”.

Logging

Log levels are configurable for all applications except keylessd.

Operations API:

The log level is configurable over the “configMap.logLevel” configuration key in the helm chart which by default is set to “info”, the following values are valid (debug, info, warning, error, critical)

Node Persistence / Circuit Storage:

The log level is configurable over the “configMap.loggingLevel.root” configuration key in the helm chart which by default is set to “info”, the following values are valid (trace, debug, info, warn, error)

PreviousDeployment InstructionsNextInfrastructure Testing

Last updated