VMware-Go-KCL – a Native Open Source Go Programming Language Library for Amazon Kinesis Data Stream Consumption – VMware Cloud Community

This post is co-authored by Connor McKelvey, Tao Jiang, Lavie Tobey, and Thiru Bhat.

VMware-Go-KCL is a native open-source Go library for Amazon Kinesis Data Stream (KDS) consumption. It allows developers to program KDS consumers in any language they want and still take advantage of the features presented by the native KDS Java API libraries. Find the project on GitHub at vmware/vmware-go-kcl.

While using KDS with cloud-native solutions, we found a gap between what is available in the community vs. what developers want. At VMware, we needed to create a solution that fits our cloud-native platform. This solution is called VMware-Go-KCL, and we’ve made it open source so any developer can use it. We also encourage ongoing contributions to keep the library current. We maintain a stable code base by thorough testing to make sure there are no bugs or regressions.

What is Amazon KDS?

Amazon Kinesis Data Streams (KDS) enables the real-time processing of streaming data at a massive scale. KDS is helpful for rapidly moving data off data producers by continuously processing and transforming it before emitting it to a data store. It also helps run real-time metrics and analytics or to derive more complex data streams for further processing.

We’ll go over some KDS design principles here, but for an excellent description of KDS and its components, see “Amazon Kinesis Data Streams Terminology and Concepts.”

KDS gains its massive throughput from its design—it’s composed of one or more shards (as shown below, the center of the diagram). Each shard can support 1 MB per second in and 2 MB per seconds out, or 1000 records per second. The put-to-get latency is under 1 second. Data is persisted across 3 AWS availability zones and stored for up to 7 days.

As shown on the right side of the diagram, one or more applications can consume the data in the Kinesis Data Streams. These are called consumers. Developers can write consumers to process the data using a native Java client library from Amazon or write the consumer in another language of their choice. Go is a popular option. However, we observed that using a language other than Java for the consumers can cause significant overhead; in addition, some developers might choose another programming language for its familiarity, usability, and portability. This is where we step in.

VMware-Go-KCL

VMware-Go-KCL brings the Go and Kubernetes communities together with a Go language native implementation of KCL (Kinesis Consumer Library) matching the same API and functional spec of the original Java KCL 2.0. In our design, we are fully compatible with the AWS Java API libraries. Using our solution, developers can write consumers in any language they choose and don’t need to worry about problems with significant overhead. There are a couple more advantages and new features, as well.

Advantages of using VMware-Go-KCL in cloud-native applications

Native Go language library with complete fidelity to Java KCL

Many cloud-native applications are written in the Go programming language, and many developers prefer to use Go loop-based workloads for ease of use, reduced memory footprint, and familiarity with other cloud-native applications written in Go.

No need to install the MultiLangDaemon process

AWS implements a Java-based daemon called MultiLangDaemon. The daemon can spawn a sub-process that runs the record processor written in any language. The MultiLangDaemon process and the record processor sub-process communicate over STDIN and STDOUT using a defined protocol. VMware-Go-KCL eliminates the need to install, run, and manage the process by providing a native Go language native library embedded into a service.

Support for the latest KDS features

Shard load balancing support via lease stealing

Lease stealing enables KCL applications to distribute Kinesis shards across multiple workers evenly. Without lease stealing, uneven shard distribution may exist between the individual workers within the KCL application, which can cause imbalanced resource utilization across workers and poses several challenges related to auto-scaling as Kinesis throughput varies throughout the day. For example, without the guarantee that new workers can acquire new shards quickly, the KCL application enters into a thrashing cycle when horizontally scaling based on a CPU threshold to handle the stream’s varying throughput. At first, the average CPU utilization of all workers peaks above a threshold, so the application is horizontally scaled. New workers cannot acquire shards and thus cannot adequately utilize the resources allocated to them, which decreases the application’s average CPU utilization below the threshold, so the application is horizontally scaled down, and the cycle repeats.

VMware-Go-KCL solves this issue using a load balancing technique referred to as lease stealing. At the end of the worker’s event loop, a rebalance workflow determines the optimal number of shards for the worker by calculating the number of known shards and active workers. The rebalance workflow decides to “claim a shard” based on the current number of shards the worker owns, the optimal number of shards per worker, and the configured maximum number of leases. Once a shard is claimed, the lease is released by the current owner, and the worker that made a claim is eligible to acquire a lease for the claimed shard in the next iteration of its event loop. This approach allows the workers in the KCL application to linearly converge on a balanced workload even during scale-up and scale-down events.

Support for developing custom consumers with dedicated throughput (enhanced fan-out)

Ilia Climpoes from CrunchyRoll contributed code and tests to support the development of custom consumers with dedicated throughput.

This feature allows each consumer to subscribe to all the shards and all the records in the stream. Data is pushed to the consumer instead of polling for data. An enhanced fan-out pipe provides up to 2MB per second of data per shard, independent of any other lines or the total number of consumers. For detailed information about enhanced fan-out, see “Developing Custom Consumers with Dedicated Throughput (Enhanced Fan-Out).”

Trying out VMware-Go-KCL

You can try out VMware-Go-KCL using the methods described at vmware/vmware-go-kcl. It needs the code in worker_test.go to be altered.

Engaging with the community and contributing to VMware-Go-KCL

We encourage the community to contribute to VMware-Go-KCL. VMware-Go-KCL is distributed under the permissive MIT license. The contribution guide details the steps and processes to contribute.