Apache Kafka offers several methods for authenticating Kafka clients to brokers. This guide aims to showcase these methods, provide guidance on enabling and configuring authentication options, and assist in determining the most suitable approach for your specific use case.
1.1: What is authentication and authorization?
Authentication is the act of confirming the identity of a user or system, ensuring that the entity interacting with the Kafka cluster is indeed who they purport to be. In contrast, authorization is the process of ascertaining the permitted actions a user or system can execute once their identity has been authenticated. This concept can be likened to entering a restaurant, where you present your identification (authentication — verifying your claimed identity) to gain entry. However, to access the exclusive VIP section, a host will verify if you have a prior reservation (authorization — determining your access privileges to a specific resource).
In the context of Kafka, authentication and authorization operate in tandem to guarantee that only authorised users can connect to the cluster and carry out designated operations.
2. Protocols supported by Apache Kafka
Kafka offers support for various authentication and authorization protocols, such as:
2.1: SASL (Simple Authentication and Security Layer)
SASL establishes a framework for incorporating authentication and data security services into connection-oriented protocols. It provides a pluggable architecture that allows different authentication mechanisms to be used, such as:
PLAIN: A simple username and password authentication mechanism.
SCRAM (Salted Challenge Response Authentication Mechanism): A more secure username and password authentication mechanism that avoids sending passwords in plaintext.
Kerberos / GSSAPI: An authentication protocol that uses tickets to allow nodes to prove their identity securely.
OAUTHBEARER: An authentication mechanism that uses OAuth 2.0 bearer tokens for authentication.
These protocols facilitate encryption and authentication for data in transit. By implementing SSL/TLS in your Kafka cluster, you can guarantee the security of all data transmitted between clients and brokers. This is achieved through the use of certificates.
SSL/TLS works by establishing an encrypted connection between the client and the broker. The client verifies the broker’s identity using the broker’s certificate, ensuring that it is communicating with a trusted entity. Additionally, the broker can require the client to present a certificate to verify the client’s identity. This mutual authentication process helps prevent unauthorised access to the Kafka cluster.
2.3: Summary
A significant distinction between TLS Client Certificates and Kerberos, as elaborated upon later, arises from the issue described in KAFKA-3700. Apache Kafka lacks native support for certificate revocation lists (CRLs), necessitating the utilisation of JVM-provided mechanisms or the employment of access control lists (ACLs) as an alternative approach to revoking access. With both SSL and SASL the following can be achieved:
Use it in conjunction with Access Control Lists (ACLs) in Kafka
Enforce identification of the client
Operate simultaneously on a protected external port and an unprotected internal port by employing traffic segregation within Kafka
3. Which protocol should you use?
The choice of authentication protocol depends on your specific requirements and environment:
SSL/TLS
Suitable for cloud environments and when you need secure communication and mutual authentication using certificates
No need to set up a dedicated server, and the authentication process is considerably less complex compared to Kerberos.
Enable the broker to verify the identity of clients (mutual authentication). Set ssl.client.auth to required.
A certificate authority (CA) is necessary to validate your certificates.
You’ll need to oversee both client and server certificates, which would be required regardless if you aimed to secure data in transit.
Unlike Kerberos, which necessitates the addition of TLS, traffic encryption is inherently integrated.
Neglecting to renew certificates with a set time-to-live (TTL) can lead to unexpected issues in the future when clients abruptly fail to authenticate.
For Kafka to function properly, SSL certificates must be loaded into the Java keystore/truststore.
Important: By default, the ssl.endpoint.identification.algorithm property is not set, resulting in the absence of hostname verification. To activate hostname verification, assign the value HTTPS to this property.
SASL/GSSAPI
Ideal for enterprise environments that already use Kerberos, such as Active Directory
Kerberos credentials offer greater flexibility and control over client authorization management and expiration.
When a Kerberos-enabled cluster is accessible from the internet, the KDC must also be exposed to the internet (on a dedicated port) and permit connections from all Kafka clients, potentially increasing security risks. This approach is strongly discouraged if Active Directory serves as the KDC.
Kerberos provides a more straightforward and adaptable method for revoking credentials, such as by modifying passwords.
Troubleshooting and resolving authentication issues can be more challenging with Kerberos.
While Kerberos itself does not offer encryption for data in transit, it can be combined with TLS/SSL to secure traffic.
Requiring clients to refresh their ticket granting tickets (TGTs) at regular intervals, such as every 10 hours, enhances security compared to using certificates with a one-year expiration. If an attacker obtains your credentials, they become invalid after the specified interval, which is not the case with SSL certificates.
SASL/PLAIN
Provides username and password authentication over a secure TLS connection.
It is easy to set up and understand compared to more complex methods like Kerberos (GSSAPI).
SASL/PLAIN allows you to plug in custom callback handlers that can verify passwords against external authentication servers or credential stores.
The key drawback of SASL/PLAIN is that the passwords are transmitted in plain text, so it is critical to only use it with TLS encryption enabled to protect the passwords on the wire. For the strongest security, consider using SASL/SCRAM instead which avoids sending plain passwords.
SASL/SCRAM
Offers enhanced security by avoiding sending plain-text passwords.
SCRAM allows for mutual authentication between the client and server.
The challenge-response authentication of SCRAM helps protect against replay attacks. It uses a one-time random nonce for each authentication attempt, making it much harder for an attacker to replay captured authentication messages.
The main drawback of SASL/SCRAM is the additional computational overhead of the challenge-response protocol, especially on the client side.
The best tip we can give you is to consider factors such as existing infrastructure, security requirements, and ease of configuration when selecting the authentication protocol.
4. Authorization in Kafka using ACLs
ACLs (Access Control Lists) allow you to control access to Kafka resources based on user or group permissions. ACLs can be applied at the cluster level, topic level, or even down to the individual operation level (read, write, create, delete)
Kafka ACLs work on the principle that Deny takes precedence over Allow. So if a user has both Deny and Allow for an operation on a resource, the Deny will be enforced.
In order to use ACLs with Kafka, you don’t need to use either SASL or TLS client certificates to identify clients. You can also use IP based client identification, but it is not secure. By default, the SSL user name will be of the form:
Principal P is [Allowed/Denied] Operation O From Host H On Resources matching ResourcePattern RP
The key components are:
Principal: The user or application being authorised, prefixed by “User:” (e.g. User:Bob)
Operation: The action being allowed or denied (e.g. Read, Write, Create, Delete, Alter, etc.)
Host: The client host from which the action is being performed
Resource: The Kafka resource being protected (e.g. Topic, Consumer Group, Cluster)
4.1: Managing ACLs
ACLs are managed using the kafka-acls.sh command line tool. Some key options include:
–add: Adds a new ACL
–remove: Removes an ACL
–list: Lists existing ACLs
–allow-principal: Specifies a principal to allow
–deny-principal: Specifies a principal to deny
–allow-host: Specifies a client host to allow
–operation: Specifies the operation to allow/deny (Read, Write, Create, etc.)
4.1: Super User and Bootstrapping
When configuring Kafka ACLs and authentication, it is recommended to designate the brokers as super users to prevent the need for creating ACLs for each broker individually. The super.users broker configuration is used to define a super user.
However, each broker must authenticate with all other brokers. When employing a SASL authentication mechanism, this requirement means that the broker’s credentials must be available before the broker starts. For instance, with SASL/PLAIN or SCRAM, the credentials should be stored in a password file or Zookeeper. In contrast, when using SASL/GSSAPI (Kerberos), the credentials must be provisioned with the KDC and included in the broker’s keytab. Consequently, it is necessary to generate broker credentials before initiating the brokers.
Alternatively, broker authentication can be performed using TLS, utilising an SSL/TLS listener instead of SASL.
If you have ACLs enabled for authorization, you would need to specify the complete distinguished name as part of the rule. In most cases, you would only require the CN attribute value (“user” in the above example) as the principal name. Users also have the ability to customise the mapping to extract attribute values.
In the previous example, if you wish to extract the value of the CN attribute, you would utilise a mapping pattern in the broker configuration like this:
If the mapping replacement is $1, then the mapped result will be “user”.
This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.
Strictly Necessary Cookies
Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.
If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.
3rd Party Cookies
This website uses Google Analytics to collect anonymous information such as the number of visitors to the site, and the most popular pages.
Keeping this cookie enabled helps us to improve our website.
Please enable Strictly Necessary Cookies first so that we can save your preferences!