Reusing Your Proto Files in Maven Multi-Module Project
It’s time for our new #BeTechReady article, and we’re glad that in this one we have a teammate from our back-end team based in Skopje. Meet Kristijan Rusu, a back-end developer who joined the Singular Team back in 2017. He started out as a QA intern, but it wasn’t long until his internship successfully resulted in a full-time hire.
It’s been 4 years since the beginning of Kristijan’s Singular Story, so we immediately knew that it would be great if he shares some of his tech knowledge with us! Kristijan, being the great teammate he is, took up the challenge and came up with an article discussing the usage of proto files in Maven multi-module projects. Interested to learn more? Read along. ⬇️
gRPC is a high-performance, open-source RPC framework. It helps in eliminating boilerplate code and helps in connecting polyglot services. Not only does gRPC use the fast HTTP/2 binary protocol, but it also uses Google’s Protocol Buffers. Protocol Buffers (protobuf for short) is a way to serialize and deserialize data, similar to JSON, but much faster and more compact. gRPC uses proto files (.proto) to define the service methods, payloads and responses, think OpenAPI and we will show you how to share those .proto files between Maven multi-module projects.
Organizing the project
There are two ways you can go about organizing the project: store all .proto files in a single module; store .proto files in a specific module for a specific API. Both ways have their pros and cons. The first option would minimize the extra project modules needed for storing the .proto files, but this also means that every client/server that needs some specific .proto file, would pull all .proto files declared and this will pollute the code.
The second will require having a separate module for each integration but would allow the client/server to pull only specific .proto files needed for integration. We opted-in for the second way as this will enable us to share additional integration files that may be needed such as DTO classes, event formats, etc. Nevertheless, the steps are the same for both types of project organization.
We will use the following .proto file for the sake of simplicity:
syntax = "proto3";
package uk.singular.example.api;
// The greeting service definition.
service Greeter {
// Sends a greeting
rpc hello (HelloRequest) returns (HelloReply);
}
// The request message containing the user's name.
message HelloRequest {
string name = 1;
}
// The response message containing the greetings
message HelloReply {
string message = 1;
}
The file is located in the API module on src/main/proto path.
example-api
|-- src
|-- main
|-- proto
|-- greeter.proto
Our project structure has the following modules:
example // Maven parent example-api // Module containing proto files example-client // Module containing client code example-server // Module containing server code
Generating code
Generating code from .proto files is fairly simple in a Java Maven project. We use Maven plugins that will automatically generate the code when running Maven lifecycle commands. This will also remove all boilerplate code needed for such integrations.
The first step in sharing the .proto files is that you need to generate the Java classes from the .proto file. You will not actually share the .proto file itself, only the generated classes.
This can be easily achieved with the following pom.xml set up for the example-api module:
<build>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>${os.maven.plugin.version}</version>
</extension>
</extensions>
<plugins>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>${protobuf.plugin.version}</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${protoc.version}:exe:${os.detected.classifier}</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}</pluginArtifact>
<protocPlugins>
<protocPlugin>
<id>quarkus-grpc-protoc-plugin</id>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-grpc-protoc-plugin</artifactId>
<version>${quarkus.plugin.version}</version>
<mainClass>io.quarkus.grpc.protoc.plugin.MutinyGrpcGenerator</mainClass>
</protocPlugin>
</protocPlugins>
</configuration>
<executions>
<execution>
<id>compile</id>
<goals>
<goal>compile</goal>
<goal>compile-custom</goal>
</goals>
</execution>
<execution>
<id>test-compile</id>
<goals>
<goal>test-compile</goal>
<goal>test-compile-custom</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
For simplicity, we added only the relevant part of the pom.xml of the module, and we defined the versions as properties to be easier to manage. This way you will generate all needed class files.
Note: If you’re not using Quarkus, just remove the quarkus-grpc-protoc-plugin from the protobuf-maven-plugin configuration.
Ensuring backward compatibility
As this is supposed to be a microservices project (the steps can also be used for monolith) important part when sharing classes between multiple services, as they can be deployed independently and have different versions on different environments is to ensure backward compatibility. Deploying one service should not break the other one.
Fortunately, Protocol Buffers have backward compatibility baked right in. gRPC can work with different versions of the .proto file, but we need to be sure that we don’t corrupt the .proto file by changing the unique number order and type of fields.
To automate validation, we will add another Maven plugin called proto-backwards-compatibility that creates a specific proto.lock file based on the .proto file. This proto.lock file is used by the same plugin to validate the change to the .proto files. The plugin will either generate a new proto.lock file if it’s missing, will validate changes of .proto files if proto.lock exists and fails the build of the module if incompatible changes were made.
To use the plugin add the following plugin to the pom.xml of the example-api module:
<plugin>
<groupId>com.salesforce.services</groupId>
<artifactId>proto-backwards-compatibility</artifactId>
<version>${protobuf.backwards.plugin.version}</version>
<executions>
<execution>
<goals>
<goal>backwards-compatibility-check</goal>
</goals>
</execution>
</executions>
</plugin>
Using the proto in a server
Using the .proto file on the server-side is as simple as adding a dependency and providing an implementation for an abstract class. For the generated code to be available, you need to run mvn install on the example-api module and add example-api as a dependency to the example-server module.
This is the code for the Mutiny service implementation:
package uk.singular.example;
import io.quarkus.grpc.GrpcService;
import io.smallrye.mutiny.Uni;
import uk.singular.example.api.GreeterOuterClass;
import uk.singular.example.api.MutinyGreeterGrpc;
@GrpcService
public class GreeterImpl extends MutinyGreeterGrpc.GreeterImplBase {
@Override
public Uni<GreeterOuterClass.HelloReply> hello(GreeterOuterClass.HelloRequest request) {
return Uni.createFrom().item(GreeterOuterClass.HelloReply.newBuilder().setMessage("Hello"+request.getName()).build());
}
}
That’s it, you’re done on the server-side.
Note: Because we’re using Quarkus, multiple abstract classes are available to be extended because Quarkus supports Mutiny, reactive and blocking implementations. If you’re not using Quarkus, only the reactive and blocking implementations will be available.
Using the proto in a client
Using the .proto file on the client-side is even simpler than using it on the server-side.
For example, using the client in a REST resource:
package uk.singular.example;
import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import io.quarkus.grpc.GrpcClient;
import uk.singular.example.api.GreeterOuterClass;
import uk.singular.example.api.MutinyGreeterGrpc;
@Path("/")
@ApplicationScoped
public class GreeterResource {
@Inject
@GrpcClient("greeter")
MutinyGreeterGrpc.MutinyGreeterStub greeter;
@GET
@Path("/hello/{name}")
public String hello(@PathParam("name") String name) {
return greeter.hello (GreeterOuterClass.HelloRequest.newBuilder().setName(name).build()).await().indefinitely().getMessage();
}
}
Note: If you’re not using Quarkus, you can check out gRPC.io for examples of how to use the generated stubs.
Further work
If you are using Quarkus like us, don’t forget to add the dependency for gRPC and add the quarkus-maven-plugin. Additionally, the same steps can be used for non-Quarkus projects. You can also remove the .proto file modules from the project to a separate repository if that suits your need better.
Extra: Simple health check for Quarkus gRPC client
package uk.singular.example;
import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;
import org.eclipse.microprofile.health.HealthCheck;
import org.eclipse.microprofile.health.HealthCheckResponse;
import org.eclipse.microprofile.health.Liveness;
import org.eclipse.microprofile.health.Readiness;
import io.grpc.Channel;
import io.grpc.ConnectivityState;
import io.grpc.ManagedChannel;
import io.quarkus.grpc.GrpcClient;
@Liveness
@Readiness
@ApplicationScoped
public class GreeterHealthCheck implements HealthCheck {
@Inject
@GrpcClient("greeter")
Channel channel;
@Override
public HealthCheckResponse call() {
ManagedChannel managedChannel = (ManagedChannel) channel;
ConnectivityState state = managedChannel.getState(true);
boolean up = ConnectivityState.READY.equals(state) || ConnectivityState.IDLE.equals(state);
return HealthCheckResponse.builder().status(up).name("greeter").withData("state",state.name()).build();
}
}
Our thoughts
We’ve shown you how to share .proto files in Maven multi-module project. We now have a consistent way to create, modify, and integrate gRPC services. Some would say this is a bad practice, but it simplifies development a lot, and managing integration points is reduced down to managing .proto files. We’ve baked backward compatibility right in the process, so you’re all covered to write your own gRPC integrations.
What an amazing read this was! Huge thanks to our Kristijan Rusu for sharing his tech knowledge with everyone. Hopefully, you will find it useful and interesting just as much as we did!
We’re always on the lookout for team players with a go-getter attitude and if you happen to be searching for a new job opportunity, check out our Career page. Maybe you will be the next person to join our team and share your knowledge in future #BeTechReady articles. 😉
