Hi all,
I'm attempting to produce AVRO messages to my Kafka topic in Confluent Cloud using the Neoload Kafka send and receive message element, but I encountered an issue where I was unable to produce.
I have two topics in Confluent Cloud, one accepting messages as strings and the other as AVRO. I've configured my Kafka send message element with all the relevant details (provided below), and for authorization and authentication, I'm using OAuth. I can successfully produce string messages to my string topic, but I'm unable to do the same for the AVRO topic. However, I also have an API key that can act as a substitute for OAuth, and with the API key, I can produce AVRO messages to the topic without any issue.
In order to produce AVRO messages, we need to specify the schema in the configuration, which I did under the parameter value.schema. Despite this, I'm still unable to produce using OAuth. The error message I'm receiving is “Sending failed: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: User is denied operation Read on Subject: com.xyxyxyx.domain.boiler_plate.model.UserInfo; error code: 40301”.
According to my developer, a crucial aspect regarding our Kafka topic is when sending AVRO messages to Kafka using OAuth authentication and utilizing the "io.confluent.kafka.serializers.KafkaAvroSerializer" for serialization, it's essential to ensure compatibility with version "7.4.0" of the serializer. Earlier versions may have limitations that could affect serialization. To guarantee seamless integration and proper serialization when sending messages to Kafka topics, we must explicitly specify version "7.4.0" of the serializer in our configuration. This ensures that our system operates smoothly and remains compatible with the required dependencies.
If this is the case, how can we pass this version along with our configuration in Neoload? If not, what could be the issue? I have all the necessary read and write access to the topic, and using IntelliJ with the same configuration details, I could successfully send AVRO messages to the topic without any issue. Only in Neoload am I facing this issue.
Configuration:
bootstrap.servers
topic
key
value : { "usernames": "Neoload_AVRO", "age": 18, "phone": "1498376382", "street": "AAAAAA", "city": "Tamilnadu", "country": "India" }
headers
security.protocol: SASL_SSL
auto.register.schemas: false
client.id
use.latest.version: true
sasl.mechanism: OAUTHBEARER
sasl.login.callback.handler.class: org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
sasl.oauthbearer.token.endpoint.url
sasl.jaas.config: org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId=' ' scope='produce' clientSecret=' ' extension_logicalCluster=' ' extension_identityPoolId=' ';
schema.registry.url:
bearer.auth.credentials.source: SASL_OAUTHBEARER_INHERIT
bearer.auth.logical.cluster:
key.serializer: org.apache.kafka.common.serialization.StringSerializer
value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
value.schema
specific.avro.reader: true
Hope I have given you all the necessary information. Thanks in advance