Overview

This page describe an overview of deploying Lenses against your Kafka clusters.

This guide walks you through manually deploying HQ and an Agent to connect to your Kafka clusters. Lenses acts as a Kafka client, it can connect to any provider exposing a Kafka-compatible API.

For more detailed guides on Helm, Docker, and Linux, see here.

How to connect to your Kafka?

To deploy Lenses against your environments, you need to:

1

Configure HQ

Optionally use your own Postgres instance.

2

Start HQ

Start HQ using your configuration from step 1.

3

Create an Environment in HQ

To connect an agent to HQ for your Kafka cluster, we need to create an environment in HQ.

4

Configure & start an Agent

To configure the agent, you need to:

1. Optionally using your own Postgres instance

2. Configure a connection to your Kafka cluster and HQ (with the key from step 3)

Prerequisites

EULA acceptance

To start HQ and an Agent, you have to accept the Lenses EULA.

For HQ, in the config.yaml set:

config.yaml

license:
  acceptEULA: true

Kafka

Any version of Apache Kafka (2.0 or newer) on-premise and in the cloud. Supported providers include:

1. Confluent Platform & Cloud

2. AWS MSK & AWS MSK Serverless

3. Aiven

4. IBM Event Streams

5. Azure HDInsight & EventHubs

Schema Registry

Any version of Confluent Schema Registry (5.5.0 or newer), APICurio (2.0 or newer) and AWS Glue.

Postgres

Only needed if you want to use your Postgres. The docker compose will start a local Postgres instance.

HQ and Agents can share the same instance, by either using a separate database or schema for HQ and each agent, depending on your networking needs.

1. Postgres server running version 9.6 or higher.

Database Role

The recommended configuration is to create a dedicated login role and database for the HQ and each Agent, setting the HQ or Agent role as the database or schema owner. Both the agent and HQ need credentials; create a role for each.

terminal

# login as superuser and add Lenses role and database
psql -U postgres -d postgres <<EOF
CREATE ROLE lenses_agent WITH LOGIN PASSWORD 'changeme';
CREATE DATABASE lenses_agent OWNER lenses_agent;

CREATE ROLE lenses_hq WITH LOGIN PASSWORD 'changeme';
CREATE DATABASE lenses_hq OWNER lenses_hq;
EOF

Networking

1. Web sockets - You may need to adjust your load balancer to allow them. See here.
2. JMX connectivity - Connectivity to JMX is optional (not required) but recommended for additional/enhanced monitoring of the Kafka Brokers and Connect Workers. Secure JMX connections are supported, including JOLOKIA and Open Metrics (MSK).

For more enable JMX for the Agent itself, see here.

Kafka ACLs

These ACLs are for the underlying Lenses Agent Kafka client. Lenses has its own set of permissions guarding access.

You can restrict the access of the Lenses Kafka client but this can reduce the functionality on offer in Lenses, e.g. not allow Lenses to create a topic at all, even though this can be managed by Lenses own IAM system.

The agent requires access to your Kafka cluster. If ACLs are enabled, you will need to allow the Agent access.

SSO (optional)

If you want to use SSO / SAML for authentication, you will need the metadata.xml file from your provider. See Authentication for more information.