Confluent Upgrades Flink Support, Advancing Real-Time Data Streaming Capabilities

Confluent, a platform that simplifies real-time data streaming and system integration, announced the latest version of its flagship product, Confluent Cloud, at the Current 2024, held this year in Austin, TX.

The event was attended by data practitioners, including developers, architects, operators, and leaders driving data applications within their organizations.

Businesses are increasingly turning to stream processing for real-time applications in machine learning (ML), predictive maintenance, personalized recommendations, and fraud detection. This approach allows organizations to seamlessly integrate and enrich their data across various functions.

Apache Kafka serves as a foundation for data streaming, while Apache Flink delivers real-time processing capabilities. Confluent builds on and integrates these technologies to provide a robust and complete solution for data streaming and processing.

Although Apache Flink has emerged as the de facto standard for stream processing, it frequently faces challenges with operational complexity, security concerns, and high infrastructure costs.

Confluent Cloud is a managed data streaming platform that simplifies data movement between applications, initially built on Apache Kafka and recently expanded to include support for Apache Flink.

The latest version of Confluent Cloud introduces key features designed to simplify and secure deployments of the Apache Flink. This includes support for the Table API, which simplifies Apache Flink for developers using Java and Python, offering flexibility to match their coding preferences and project needs.

The support for Table API enables users to streamline the coding process by using their preferred Integrated Development Environment (IDE) with auto-completion, refactoring tools, and compile-time checks to enhance code quality, reduce runtime issues, and simplify debugging.

Support for the Table API is currently in open preview for testing and experimentation, with general availability coming soon.

The latest upgrade to Confluent Cloud introduces private networking support for Apache Flink, enhancing the security of data streaming operations. As data volumes grow and cloud and hybrid solutions become more prevalent, protecting against unauthorized access and cyber threats has never been more crucial.

The new network configuration process enables companies to protect in-transit data and merge multiple streams within a secured environment. This upgrade could pave the way for organizations in highly regulated industries, such as healthcare companies, to start streaming data.

Private networking support is currently available on AWS for enterprise and dedicated clusters, with plans to expand to additional cloud platforms in the near future.

Teams using real-time data platforms like Apache Kafka often face challenges with fragmented tools and inefficient workflows, making it difficult to integrate data into applications. Confluent has introduced an extension for Visual Studio Code (VS) to streamline workloads and accelerate development cycles. With 54% of developers globally using VS Code, this new extension promises to enhance the development experience by providing a more integrated and efficient process.

Key features of the extension include simplifying topic management, enabling integrated coding and debugging, and efficiently managing cloud resources—all within VS Code. The Confluent Extension for Visual Studio Code supports Kafka clients and is currently available in early access. General availability is expected later in 2024.

To enhance the security of the platform, Confluent has debuted Client-Side Field Level Encryption to safeguard most sensitive data while meeting regulatory requirements.

Users can encrypt individual fields within data streams, ensuring that even if unauthorized parties access the data, they will not be able to decipher it.

Users also get granular access control to choose which specific fields to encrypt. This feature also reduces the cost of ownership by reducing the need for topic duplication. Client-Side Field Level Encryption is set to become generally available to all Confluent Cloud customers soon.

According to Stewart Bond, Research Vice President at International Data Corp., harnessing Apache Flink’s capabilities for stream processing allows developers to build applications that can analyze and react to real-time data with remarkable speed. This dramatically boosts responsiveness and improves the overall user experience.

“Managed Apache Flink solutions can eliminate the complexities of infrastructure management while saving time and resources,” stated Steward Bond. “Businesses must look for a Flink solution that seamlessly integrates with the tools, programming languages, and data formats they’re already using for easy implementation into business workflows.”