undefined kafka components for Go kafka

Question

I was trying to install one of my go files. But I bumped into this error

C:\mygoproject>go install kafkapublisher.go

\#command-line-arguments
.\kafkapublisher.go:8:65: undefined: kafka.Message

.\kafkapublisher.go:10:19: undefined: kafka.NewProducer

.\kafkapublisher.go:10:38: undefined: kafka.ConfigMap

.\kafkapublisher.go:17:31: undefined: kafka.Event

.\kafkapublisher.go:19:26: undefined: kafka.Message

On my kafkapublisher.go file, I already imported the kafka dependency:

    import (
        "github.com/confluentinc/confluent-kafka-go/kafka"
        "log"
    )

even on my go.mod file

    module mymodule
    
    go 1.12
    
    require (
        github.com/aws/aws-lambda-go v1.15.0
        github.com/confluentinc/confluent-kafka-go v1.3.0
    )

I followed this documentation: https://docs.confluent.io/current/clients/go.html

screenshot

Kinetic

Answer 1

I was facing the same issues.

Kafka Go client is based on the C library. So setting flag CGO_ENABLED=1 will enable go to use C libraries for kafka client.

Hope it saves someone's time.

Vano

Thank you! saved my skin, I was trying to build a go project that used confluent kafka connector in an alpine-go container. The error message was rather vague. – Torticollis

Answer 2

When you build your image, use -tags musl (for alpine linux, @see github.com/confluentinc/confluent-kafka-go) and active CGO_ENABLED to activate lib in C because Kafka Go client is based on the C library librdkafka

In dockerfile :

FROM golang:1.16-alpine as builder

ARG GIT_TAG_NAME

RUN apk --no-cache update && \
apk --no-cache add git gcc libc-dev

# Kafka Go client is based on the C library librdkafka
ENV CGO_ENABLED 1
ENV GOFLAGS -mod=vendor
ENV GOOS=linux
ENV GOARCH=amd64

RUN export GO111MODULE=on

RUN go build -tags musl -ldflags "-s -w -X main.Version=$GIT_TAG_NAME" -o bin/main ./cmd/main

Note: -tags musl : if you want to use the built-in librdkafka -tags dynamic : if you want to use your own librdkafka.

Hagride

can you clarify the reasons to use of buildkit ? And explain why you recommended to use the musl tag during the build of the binary ? – Festival

done, I remove buildkit (for cache docker), and i explain why use musl tag – Hagride

indeed, downloading gcc and building with -tags musl is required to get the build going. – Torticollis

Answer 3

I already figured out this one. I installed Confluent's Kafka Go Client. Instructions are here: https://docs.confluent.io/current/clients/go.html#

The library is not supported on windows though, so I had to use virtual machine (Oracle VM Box) to build and run my code.

I also needed to compile and install librdkafka before installing the Confluent's GO Kafka Client: https://github.com/confluentinc/confluent-kafka-go/blob/master/README.md

Thanks.

Kinetic

Thanks for this, this put me in right direction. I created another answer with exact steps. https://mcmap.net/q/1124248/-undefined-kafka-components-for-go-kafka – Ostracon

Answer 4

• Make sure to activate CGO_ENABLED -> CGO_ENABLED=1.
• CGO_ENABLED is an environment variable in the Go programming language that controls whether the Go compiler includes support for calling C code.

Begrudge

Answer 5

If you're on Windows, follow these steps

1. Install GCC. I installed from here https://jmeubank.github.io/tdm-gcc/download/
2. Restart terminal
3. Run SET CGO_ENABLED=1
4. Run the failing command again.

Ostracon

Answer 6

clearly dependencies are not being imported,

if you run the go build command, it will download the necessary dependencies and compile the code etc

try running go build ./...

Illiberal

Answer 7

The dependencies have not been downloaded.

You can either use a go get to download the package. Or Use

1. go mod download
2. go mod tidy
3. go mod vendor - This will create a vendor folder with the required dependencies

Barocchio

Run your application and send me screenshot of the error. Try go run. – Barocchio

please refer to link on my description above. labeled "screenshot" – Kinetic

Did you allow these steps? git clone github.com/edenhill/librdkafka.git cd librdkafka ./configure --prefix /usr make sudo make install – Barocchio

You should install librdkafka on your OS. – Barocchio

im using windows. can i compile using windows? i was able to compile on windows using set GOOS=linux, but im not running this on windows. i deploy this on aws lambda/. – Kinetic

If you are trying to run this locally. confluent-kafka-go is not supported on Windows. Says on the README.md as well. If your deploying this to AWS lamda linux. Make sure you install librdkafka on it. If you try running it locally it fails. – Barocchio

confluent-kafka-go is a wrapper around librdkafka. So you got to install it. – Barocchio