Kubernetes Deployment

The Components
The method Keyless adopts to ship its Infrastructure Components is via Helm Chart deployment.
The following diagram illustrates a general overview of the Infrastructure Components
Here's an example diagram featuring Openshift as the K8s flavour
The Keyless Infrastructure is composed of the following components:
Keyless Node: responsible for implementing the distributed privacy-preserving protocol for Keyless zero-proof biometric authentication
Node Persistence Service: service implementing a database abstraction layer (Postgres and OracleDB currently supported) 
Database: database used by Keyless to store user data 
Circuit Storage Service: service offering internal-use REST APIs to allow Keyless node to store circuits needed to authenticate users 
Database + S3 Storage: used by Circuit Storage Service to store circuits and metadata 
Keyless Rest API (Operations API): offers REST APIs for transaction confirmation and other backend services.
Each of the Keyless components has it's own chart and can be deployed standalone, but we strongly suggest to use our Main Keylessd Helm Chart to deploy all needed components as one, more on this in the Deployment Steps section
Access the Repositories
The customer is going to use 2 repositories: 
A Docker Image Registry: quay.io/keyless.technologies 
A Helm Chart Registry: keylesstech.github.io
To access the Docker repository the following command needs to be executed: 
$ docker login quay.io -u < username > -p < password >
There is no need for authentication against the Helm Chart Registry.
Deployment Steps
The first step is to add our helm repository to your helm repository list with the following command
$ helm repo add keyless https://keylesstech.github.io
Then you can list the charts 
$ helm search repo keyless
And get the values.yaml from the keylessd chart, which is our global one. 
$ helm show values keyless/keylessd
Our suggestion, to perform a deployment, is to include our main helm chart, keylessd as a dependency of your own chart so that you can decide how to provide secret creation and other dependencies that you may have. ( for example you may decide to deploy custom monitoring agents alongside keylessd).
Once you’re satisfied with your configuration you can simply apply it with 
$ helm upgrade --install <release-name (for example keylessd)> keyless/keylessd
Configuration
The Deployment configuration is customizable using the helm values.yaml file which is divided into the following sections: 
global: {}
​
keylessd:
​
  circuit-storage:
    ...
  node:
    ...
  operations-api:
    ...
  node-persistence:
    ...
global : global values such as :
namespace: namespace name where to deploy the resources ( optional ), defaults to default 
token: to authenticate to the registry ( optional )
createNamespace: to create the namespace if it doesn't already exists
circuit-storage: configuration specific to Circuit Storage Service 
node: configuration specific to Keyless Node 
operations-api: configuration for the Operation API component 
node-persistence: configuration for the Node Persistence API component
Global
As stated above, the global section is optional and it is used if you do not desire to deploy each component into its own namespace but prefer to have all in a single namespace. You will then need to provide the name of the namespace and to set createNamespace variable to true as shown below
global:
  token: # eyJhd.....................
  namespace: keyless
  createNamespace: true
​
global.token variable is used if you want our chart to take care of creating the imagePullSecret secret, used by the Deployment to access the docker registry, usually when the intention is to deploy each service into its separate namespace. 
​
​
Infact as you may know, to be able to reference a secret used to pull docker images, that secret needs to be created inside each namespace. 
Otherwise you will be tasked with creating the secret registry of type kubernetes.io/dockerconfigjson so that it can be accessed by the Deployment. 
To know more about the format of the secret registry please refere to the following link
https://kubernetes.io/docs/concepts/configuration/secret/#docker-config-secrets
If your registry doesn't require authentication, ( for example if syncronization between our Docker Registry and yours is already set up ) then the token variable is not needed. 
Circuit Storage Service
As this service is tasked with reading/writing from an s3-compatible storage it will require access key and secret key to be available. It will also require a database to hold metadata. If you want to supply the corresponding Secret to hold the credentials, skip this section.
Expose Circuit Storage Service
keylessd:
  circuit-storage:
    publicRoute: true
    createRoute: true
    createIngress: false
    host: circuit-storage
    domain: keyless.local
To expose Circuit Storage service outside the Cluster publicRoute variable may be used.
In addition to this, it is possible to specify if Kubernetes should create an Ingress or a Route , depending if you're running on a vanilla cluster or on Openshift.
As Circuit Storage service is only a backend service, the default values are all false.
In case the values are set to true you will also need to set the following two variables: 
host
domain 
So that the corresponding Ingress/Route Object will be configured to get traffic from <host>.<domain>.
Custom Certificate to Access the Database or the S3-Compatible Backend
keylessd:
  circuit-storage:
    customCA: true
    certificates:
      - name: storage-cert
        mountPath: "/etc/ssl/custom-ca/storage-cert.pem"
        subPath: "storage-cert.pem"
        readOnly: true
      - name: db-cert
        mountPath: "/etc/ssl/custom-ca/db-cert.pem"
        subPath: "db-cert.pem"
        readOnly: true
We support custom CA certificates in case it’s needed, setting the customCA variable to true and adding the following ConfigMap object in your chart
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: storage-cert
  namespace: my-namespace-name
data:
  storage-cert.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: db-cert
  namespace: my-namespace-name
data:
  db-cert.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
This could be used in case the s3-compatible backend exposes a secure connection with a certificate or if the database requires additional authentication requirements.
The certificates will be mounted to a specific location in the pod and added at runtime in the internal keystore of our application.
The mountPath  must always be in the format /etc/ssl/custom-ca/<certificate>.pem
Exposed Java SpringBoot Endpoints
The endpoints section of the values.yaml file, allows you to customize the amount of endpoints exposed by Circuit Storage, usually the defaults are set to expose env variables ( passwords are not shown), metrics and health-check.
keylessd:
  circuit-storage:
    configMap:
      endpoints:
        web: health,env,metrics
        metrics: true
        health: true
        env: true
        ...
Datadog Integration
We support Datadog Integration for Circuit Storage by setting up the datadogApiKey in the secrets section and the datadog.uri ( which may vary depending on whether you use datadog.eu, datadog.com etc.)
keylessd:
  circuit-storage:
    configMap:
      datadog:
        uri: https://api.datadoghq.eu
​
    secrets:
      datadogApiKey: "..."
In case you do not wish to use this integration, leave blank
Adding Java Options to Circuit Storage
The javaOpts variable allows the customization of java options which may need to be adjusted according to particular needs.
keylessd:
  circuit-storage:
    configMap:
      javaOpts: >-
        -XX:MaxRAMPercentage=75.0
        -Dspring.profiles.active=oracle,onprem
        -Dlogging.level.com.amazonaws=DEBUG
Database Connection
configMap.springDatasource.url variable holds the jdbc-formatted string to allow Circuit Storage to connect to its own database. The Instance needs a dedicated schema to be available ( default  “circuit-storage”)
keylessd:
  circuit-storage:
    configMap:
      springDatasource:
        # Oracle
        url: jdbc:oracle:thin:@//<hostname>:1521/<service>
        # or Postgres
        url: jdbc:postgresql://<hostname>:5432/<dbName>
Database Initialization
To manage the database schema Circuit Storage makes use of the Flyway library.
keylessd:
  circuit-storage:
    configMap:
      springFlyway:
        enabled: true
        locations: oracle
configMap.springFlyway.enabled variable allows you to enable or disable database initialization scripts to be run at start time. 
configMap.springFlyway.locations variable allows you to set “oracle” or “postgres” location depending on which database you’re using ( currently only the two mentioned are supported).
Disabling Flyway means that you'll need to initalize the database yourself, so please, if that's the case, reach out to us so that we can provide you the database initialization script to you
Mutual TLS Support for Oracle Database
Starting with release 0.1.4 it is possible to connect to Oracle Database using mutual TLS authentication.
To achive this the Oracle Wallet archive is needed.
To configure Circuit Storage to use Oracle Wallet follow this procedure:
Unzip the Oracle Wallet
unzip wallet.zip -d mywallet
Archive:  wallet.zip
  inflating: mywallet/ewallet.pem    
  inflating: mywallet/README         
  inflating: mywallet/cwallet.sso    
  inflating: mywallet/tnsnames.ora   
  inflating: mywallet/truststore.jks  
  inflating: mywallet/ojdbc.properties  
  inflating: mywallet/sqlnet.ora     
  inflating: mywallet/ewallet.p12    
  inflating: mywallet/keystore.jks
Execute the following command to create a Secret containing all files inside the wallet.
In the example below the name of the secret will be "wallet"
$ kubectl create secret generic wallet --from-file=./mywallet/ -n <namespace-name>
secret/wallet created
Modify values.yaml, given the following tnsnames.ora example file
tnsnames.ora
circuitstoragetest_high = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
circuitstoragetest_low = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
circuitstoragetest_medium = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
circuitstoragetest_tp = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
circuitstoragetest_tpurgent = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
and TNS_ADMIN=/network/admin as the directory which the wallet secret will be mounted into, as such:
keylessd:
  circuit-storage:
    mtls:
      secretName: wallet
    configMap:
      springDatasource:
        url: jdbc:oracle:thin:@circuitstoragetest_high?TNS_ADMIN=/network/admin
Remember that you will still need to provide database username and password as the following secrets:
secrets.datasourceUsername
secrets.datasourcePassword
Override Circuit Storage Resource Name
It is possible to override the default resource name of Circuit Storage using the fullnameOverride variable. Otherwise the resource name assumed will be “circuit-storage-<release-name>”.
Keyless Node
Expose Keyless Node Service
keylessd:
  node:
    publicRoute: true
    createRoute: false
    createIngress: true
    host: node
    domain: keyless.technology
To expose Keyless Node service outside the Cluster, publicRoute variable needs be used.
In addition to this, it is possible to specify if Kubernetes should create an Ingress or a Route , depending if you're running on a vanilla cluster or on Openshift.
As Keylessd Node needs to be exposed the default values for publicRoute, createRoute and createIngress are as shown above.
You will also need to set the following two variables: 
host
domain 
as shown above, so that the corresponding Ingress/Route Object will be configured to get traffic from <host>.<domain>.
As there can be edge cases where you would prefer to manage Ingresses outside our Chart it is possible to set publicRoute to false
In this case you'll need to provide the Ingress/Route configuration, but keep in mind that the traffic needs to flow from the Ingress controller to the Keyless Node in passthrough mode as the TLS termination happends at the node level. 
As the Keyless node will use a certificate that will be mounted at runtime into the pod, to terminate the TLS connection, regardless of the value of publicRoute, it is required to specify the host and domain to be used, which need to corresponds to ones used to generate the certificate.
As an example, let’s say that our final endpoint is node.keyless.technology, and that we have a certificate generated using the correct CN (again node.keyless.technology) then in this case the host variable will correspond to "node", meanwhile domain will correspond to “keyless.technology”.
keylessd:
  node:
    host: node
    domain: keyless.technology
        
certName variable is used to reference the secret name which hosts the certificate.
keylessd:
  node:
    certName: node
The expected format for the certificate is the one refecenced here: https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets
configMap.nodeConfig.storageUrl variable is used to reference Circuit Storage service in the format http://<service-name>.<namespace>.svc.cluster.local
configMap.nodeConfig.databaseUrl variable is used to reference Node Persitence service in the format http://<service-name>.<namespace>.svc.cluster.local
keylessd:
  node:
    configMap:
      nodeConfig:
        storageUrl: http://circuit-storage-keylessd.keyless.svc.cluster.local
        databaseUrl: http://node-persistence-keylessd.keyless.svc.cluster.local
Override Keyless Node Resource Name
It is possible to override the default resource name of Keyless Node using the fullnameOverride variable. Otherwise the resource name assumed will be “node-<release-name>”.
Operations API
Expose Operations API Service
keylessd:
  operations-api:
    publicRoute: true
    createRoute: false
    createIngress: true
    host: operations-api
    domain: keyless.local
To expose Operations API service outside the Cluster, publicRoute variable can be used.
In addition to this, it is possible to specify if Kubernetes should create an Ingress or a Route , depending if you're running on a vanilla cluster or on Openshift.
As Operations API needs to be exposed the default values for publicRoute, createRoute and createIngress are as shown above.
You will also need to set the following two variables: 
host
domain 
as shown above, so that the corresponding Ingress/Route Object will be configured to get traffic from <host>.<domain>.
As there can be edge cases where you would prefer to manage Ingresses outside our Chart it is possible to set publicRoute to false
In this case you'll need to provide the Ingress/Route configuration, but keep in mind that the traffic needs to flow from the Ingress controller to Operations API in edge mode as the TLS termination happends at the cluster level. 
configMap.nodePersistenceApiUrl variables a is used to reference NodePersitence service in the format http://<service-name>.<namespace>.svc.cluster.local
keylessd:
  operations-api:
    configMap:
      nodePersistenceApiUrl: https://node-persistence.default.svc.cluster.local:8080
Override Operations API Resource Name
It is possible to override the default resource name of Operations API using the fullnameOverride variable. Otherwise the resource name assumed will be “operations-api-<release-name>”.
Node Persistence
This service is tasked with reading/writing from/to a database. It will require a secret to be able access it.
Exposed Java SpringBoot Endpoints
The endpoints section of the values.yaml file, allows you to customize the amount of endpoints exposed by Node Persistence, usually the defaults are set to expose env variables ( passwords are not shown), metrics and health-check.
keylessd:
  node-persistence:
    configMap:
      endpoints:
        web: health,env,metrics
        metrics: true
        health: true
        env: true
        ...
Datadog Integration
We support Datadog Integration for Node Persistence by setting up the datadogApiKey in the secrets section and the datadog.uri ( which may vary depending on whether you use datadog.eu, datadog.com etc.)
keylessd:
  node-persistence:
    configMap:
      datadog:
        uri: https://api.datadoghq.eu
​
    secrets:
      datadogApiKey: "..."
In case you do not wish to use this integration, leave blank
Adding Java Options to Node Persistence
The javaOpts variable allows the customization of java options which may need to be adjusted according to particular needs.
keylessd:
  node-persistence:
    configMap:
      javaOpts: >-
        -XX:MaxRAMPercentage=75.0
        -Dlogging.level.com.amazonaws=DEBUG
Setting The Database Type
At the moment of writing Node Persistence supports Oracle and Postgres databases
To set which database type Node Persistence will need to use, it is possible to set springProfile variable to either "oracle" or "postgres"
keylessd:
  node-persistence:
    configMap:
      springProfile: oracle
Database Connection
configMap.springDatasource.url variable holds the jdbc-formatted string to allow Node Persistence to connect to its own database. The Instance needs a dedicated schema to be available ( default  “keylessd”)
keylessd:
  node-persistence:
    configMap:
      springDatasource:
        # Oracle
        url: jdbc:oracle:thin:@//<hostname>:1521/<service>
        # or Postgres
        url: jdbc:postgresql://<hsotname>:5432/<dbName>
Database Initialization
To manage the database schema Node Persistence makes use of the Flyway library.
keylessd:
  node-persistence:
    configMap:
      springFlyway:
        enabled: true
        locations: oracle
configMap.springFlyway.enabled variable allows you to enable or disable database initialization scripts to be run at start time. 
configMap.springFlyway.locations variable allows you to set “oracle” or “postgres” location depending on which database you’re using ( currently only the two mentioned are supported).
Disabling Flyway means that you'll need to initalize the database yourself, so please, if that's the case, reach out to us so that we can provide you the database initialization script to you
​
Mutual TLS Support for Oracle Database
Starting with release 0.1.4 it is possible to connect to Oracle Database using mutual TLS authentication.
To achive this the Oracle Wallet archive is needed.
To configure Node Persitence to use Oracle Wallet follow this procedure:
Unzip the Oracle Wallet
unzip wallet.zip -d mywallet
Archive:  wallet.zip
  inflating: mywallet/ewallet.pem    
  inflating: mywallet/README         
  inflating: mywallet/cwallet.sso    
  inflating: mywallet/tnsnames.ora   
  inflating: mywallet/truststore.jks  
  inflating: mywallet/ojdbc.properties  
  inflating: mywallet/sqlnet.ora     
  inflating: mywallet/ewallet.p12    
  inflating: mywallet/keystore.jks
Execute the following command to create a Secret containing all files inside the wallet.
In the example below the name of the secret will be "wallet"
$ kubectl create secret generic wallet --from-file=./mywallet/ -n <namespace-name>
secret/wallet created
Modify values.yaml, given the following tnsnames.ora example file
tnsnames.ora
nodepersistencetest_high = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
nodepersistencetest_low = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
nodepersistencetest_medium = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
nodepersistencetest_tp = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
nodepersistencetest_tpurgent = (description= (address=(protocol=...)(port=...)(host=...))(connect_data=(service_name=...))(security=(ssl_server_cert_dn="CN=..., OU=..., O..., L..., ST=..., C=..")))
and TNS_ADMIN=/network/admin as the directory which the wallet secret will be mounted into, as such:
keylessd:
  node-persistence:
    mtls:
      secretName: wallet
    configMap:
      springDatasource:
        url: jdbc:oracle:thin:@nodepersistencetest_high?TNS_ADMIN=/network/admin
Remember that you will still need to provide database username and password as the following secrets:
secrets.datasourceUsername
secrets.datasourcePassword
Override Node Persistence Resource Name
It is possible to override the default resource name of Node Persistence using the fullnameOverride variable. Otherwise the resource name assumed will be “node-persistence-<release-name>”.
Logging
Log levels are configurable for all applications except keylessd.
Operations API:
The log level is configurable over the “configMap.logLevel” configuration key in the helm chart which by default is set to “info”, the following values are valid (debug, info, warning, error, critical)
Node Persistence / Circuit Storage:
The log level is configurable over the “configMap.loggingLevel.root” configuration key in the helm chart which by default is set to “info”, the following values are valid (trace, debug, info, warn, error)
Last modified 10mo ago