gRPC 104: Pagination, Authentication & Interceptors


3.7k words • 22 min read
backend › grpc
, , , , , , ,

In the last post, we have achieved a good milestone of using a real datastore for storing and retrieving the records that the users are creating. We have already covered the services of users and todos for various CRUD operations.

This post will focus on polishing the application we have already built so that we can tighten the loose ends we left behind while covering the essentials.

The previous articles in this series:

  1. gRPC vs REST
  2. Introduction to Protocol Buffers
  3. Protocol Buffers Compilation & Serialization
  4. Introduction to gRPC
  5. gRPC 101: Creating Services
  6. gRPC 102: Creating Clients
  7. gRPC 103: Connecting to MySQL Database
gRPC with it's mascot
gRPC with it's mascot

This post will talk about some of the essential and some slightly advanced gRPC and web development concepts like the need for pagination, the importance of authentication and how interceptors make complex task easy. So let’s begin this journey.

Pagination

If you are completely new to web development then the concept of pagination simply means that we try to fetch records in batches or chunks. Suppose a user is using our applications for years and during his tenure he has more than thousands of todos.

In such scenarios it’s not a wise idea to send all the todos at once in the FindAll or GetAll unary RPCs. This is because the data volume will be too high which can cause load on the database, the frontend application (like web or mobile) which is fetching the data.

More on pagination techniques can be read here: Pagination Techniques for APIs

Pagination in gRPC

Google has a guideline for the pagination for RPCs, here’s the link API 158.

Since adding pagination during later stages of the application is backwards incompatible, we should ideally support pagination from the starting of the listing RPCs. This may sound like an anti-pattern in our series of blogs but since our application has not live users this is okay for us.

We will be using a cursor pagination where the cursor is actually the created_at field of the todo.

ProtoBuf Update

To allow pagination, we need to update both the Listing RPCs of the Todo and User proto file. For the file protos/todo.proto the changes will be: 

1
2
3
4
5
6
7
8
9
message ListTodosRequest {
  int32 page_size = 1;
  string page_token = 2;
}

message ListTodosResponse {
  repeated Todo todos = 1;
  string next_page_token = 2;
}

Note: Make sure you compile the proto files after the changes by calling make from the terminal.

Service Update

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
func (ts *TodoService) ListTodos(_ context.Context, req *pb.ListTodosRequest) 
(*pb.ListTodosResponse, error) {
    // check if req has valid params
    size := req.GetPageSize()
    if size == 0 {
        size = 10
    }
    if size > 1000 {
        size = 1000
    }
    if size < 0 {
        return nil, status.Error(codes.InvalidArgument, "page size must be positive")
    }

    // token will be the created_at of the last todo in the previous page
    token := req.GetPageToken()
    last_created_at := time.Now()
    var err error
    if token != "" {
        last_created_at, err = time.Parse(time.RFC3339, token)

        if err != nil {
            return nil, status.Error(codes.InvalidArgument, "invalid page token")
        }
    }

    // get all from db
    todos, err := ts.repo.FindAll(last_created_at, size)
    if err != nil {
        log.Printf("error listing todos: %+v", err)
        return nil, status.Error(codes.Internal, "something broke - find all todos")
    }

    // return response
    return &pb.ListTodosResponse{
        Todos:         todos,
        NextPageToken: todos[len(todos)-1].GetCreatedAt().AsTime().Format(time.RFC3339),
    }, nil
}

There are multiple changes which we have done in the service RPC for the ListTodos. let’s give a walkthrough of the changes:

  • Accepting the request for page_size and page_token.
  • Validating that the size is not invalid with proper gRPC errors
  • Passing the size and token to the repository method.
  • Handling the time format for the token.
  • Returning the next_page_token for the front-end to use for subsequent calls

Now from the grpcui, we can pass the page size and also the page token for our listing RPC calls.

listing RCP with page size & token
Updated listing RCP with page size and page token

Repository Updates

We just saw that we have to pass the changes to the MySQL repository so that the database can return adequate response for the requested data. Here’s the repository changes we have to do: 

1
2
3
4
5
6
7
type TodoRepository interface {
    Save(todo *pb.Todo) (*pb.Todo, error)
    FindByID(id string) (*pb.Todo, error)
    FindAll(last_created_at time.Time, page_size int32) ([]*pb.Todo, error)
    Delete(id string) error
    UpdateTodo(todo *pb.Todo) error
}

And the FindAll method implementation is as follows: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
func (r *MySQLTodoRepository) FindAll(last_created_at time.Time, size int32) 
([]*pb.Todo, error) {
    query := "SELECT uuid, title, description, is_done, priority, created_at FROM todos"

    if !last_created_at.IsZero() {
        query += " WHERE created_at < '" + last_created_at.Format("2006-01-02 15:04:05") + "'"
    }

    query += " ORDER BY created_at DESC"

    if size > 0 {
        query += " LIMIT " + strconv.Itoa(int(size))
    }

    rows, err := r.db.Query(query)
    if err != nil {
        log.Printf("error finding todos: %+v", err)
        return nil, err
    }
    defer rows.Close()

    var todos []*pb.Todo
    for rows.Next() {
        // holders
        var todo pb.Todo
        var priority string
        var createdAt time.Time

        if err := rows.Scan(
            &todo.Uuid, &todo.Title, &todo.Description, &todo.Done,
            &priority, &createdAt,
        ); err != nil {
            return nil, err
        }

        // attach createdAt
        todo.CreatedAt = timestamppb.New(createdAt)

        switch priority {
        case "low":
            todo.Priority = pb.Todo_PRIORITY_LOW
        case "medium":
            todo.Priority = pb.Todo_PRIORITY_MEDIUM
        case "high":
            todo.Priority = pb.Todo_PRIORITY_HIGH
        default:
            todo.Priority = pb.Todo_PRIORITY_LOW
        }

        todos = append(todos, &todo)
    }

    return todos, nil
}

The prominent changes we made are:

  • Accepting the page size and the token in the arguments
  • Using the values with conditional checks for query construction
  • Make sure to properly order the LIMIT and ORDER BY and converting the size to Integer

Everything else remains the same. Now that we have the pagination in place we will move to the security aspect of the application and let’s explore authentication using JWT.

Authentication

Currently our app is open for all and for everything.

What I mean to say to is that suppose there are two users John and Johnny, thy will register themselves and start adding data to the application. Surprisingly they can not only see each other’s data they can also manipulate the data of one another. There’s no boundary no validation whether the data being manipulated is actually owned by the manipulator or not.

Thant’s when we need Authentication. It’s a boundary that can bring in certain restrictions of data usages and manipulations.

If you are already familiar with REST APIs you might have encountered JWT or JSON Web Token already.

JWT

A JWT is a compact token that represents a set of claims (data) which is stateless and is signed usually with a secret or private key. The format of a JWT string token usually has three parts and looks like: header.payload.signature

Using JWT

We will create a JWT token when the user will try to login after registration.

Notes

  • We are not using passwords or OTP for login, for now we will demonstrate login using email or username, if it is present in the database we will allow login and provide them the JWT token on successful login.
  • It’s a good exercise for you to add support for passwords for the users. But be careful to not store plain password in the database. Store the hashed password in the database.

Let’s add a file lib/jwt.go which will contain the code to create a token using uuid of the user and also to validate the token. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
package lib

import (
    "errors"
    "fmt"
    "os"
    "time"

    "github.com/golang-jwt/jwt/v4"
)

var jwt_secret = []byte(getJWTSecret())

func getJWTSecret() string {
    secret := os.Getenv("JWT_SECRET")
    if secret == "" {
        secret = "random-secret"
    }
    return secret
}

func GenerateJWT(uuid string) (string, error) {
    // claim map
    claims := jwt.MapClaims{
        "uuid": uuid,
        "exp":  time.Now().Add(24 * time.Hour).Unix(),
        "iat":  time.Now().Unix(),
    }

    token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
    return token.SignedString(jwt_secret)
}

func ValidateJWT(tokenString string) (string, error) {
    token, err := jwt.Parse(tokenString, func(token *jwt.Token) (interface{}, error) {
        // Ensure HMAC signing method
        if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
            return nil, fmt.Errorf("unexpected signing method: %v", token.Header["alg"])
        }
        return jwt_secret, nil
    })

    if err != nil {
        return "", err
    }

    claims, ok := token.Claims.(jwt.MapClaims)
    if !ok || !token.Valid {
        return "", errors.New("invalid token")
    }

    uuid, ok := claims["uuid"].(string)
    if !ok {
        return "", errors.New("uuid claim missing or invalid")
    }

    return uuid, nil
}

There are a few things about the code above:

  • The function GenerateJWT will generate a JWT token using the uuid of the user
  • The function ValidateJWT will take an existing token and validate it
  • A JWT token has claim, the algorithm for encryption and the encryption secret.
  • We are taking the sensitive token secret from the env.

We will use the GenerateJWT in the user service particularly when the user is trying to login.

Updating User Login

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
func (us *UserService) Login(_ context.Context, req *pb.LoginUserRequest) 
(*pb.LoginUserResponse, error) {
    if req.GetEmail() == "" {
        return nil, status.Error(codes.InvalidArgument, "missing email of user")
    }

    user, err := us.repo.FindByEmail(req.GetEmail())
    if err != nil {
        log.Printf("error finding user: %+v", err)
        return nil, status.Error(codes.Internal, "something broke while finding user")
    }

    jwt_token, err := lib.GenerateJWT(user.Uuid)
    if err != nil {
        log.Printf("error generating JWT token: %+v", err)
        return nil, status.Error(codes.Internal, "failed to generate JWT token")
    }

    return &pb.LoginUserResponse{
        Token: jwt_token,
        Email: user.Email,
    }, nil
}

Interceptors

So now we have the JWT which we can use to check if the user who is calling the RPC is a valid register user of the system. How can we do that?

We need something like a middle man who can intercept the RPC call and validate that it is a valid call before it hit’s the business logic and the data layer. This middleman concept in the gRPC framework is called interceptor.

In Node.js and RubyOnRails, we call them middlewares and in a few frameworks we also call them filters.

Let’s write an interceptor function to check if the requested RPC call is having a JWT token in it and it’s a valid token which can yield us a valid uuid.

Request Metadata

We have already seen the Metadata section when invoking a RPC call via grpcui. Let’s explore more about it.

Metadata is a side channel that allows clients and servers to provide information to each other that is associated with an RPC. gRPC metadata is a key-value pair of data that is sent with initial or final gRPC requests or responses. It is used to provide additional information about the call, such as authentication credentials, tracing information, or custom headers.

We will add the JWT token received after the Login RPC call in the Request Metadata of the subsequent RPC calls.

The JWT token added in the Request Metadata in grpcui
The JWT token added in the Request Metadata in grpcui

Interceptor Function

Now we will write the interceptor function which will validate the JWT token and also it will extract and add the uuid of the user in the context. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
package lib

import (
	"context"
	"strings"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/metadata"
	"google.golang.org/grpc/status"
)

type contextKey string

const UserUUIDKey contextKey = "user_uuid"

func JWTInterceptor() grpc.UnaryServerInterceptor {
    return func(
        ctx context.Context,
        req interface{},
        info *grpc.UnaryServerInfo,
        handler grpc.UnaryHandler,
    ) (interface{}, error) {

        md, ok := metadata.FromIncomingContext(ctx)
        if !ok {
            return nil, status.Error(codes.Unauthenticated, "missing metadata")
        }

        authHeader := md.Get("authorization")
        if len(authHeader) == 0 {
            return nil, status.Error(codes.Unauthenticated, "missing authorization header")
        }

        token := authHeader[0]
        if !strings.HasPrefix(token, "Bearer ") {
            return nil, status.Error(codes.Unauthenticated, "invalid authorization format")
        }

        token = strings.TrimPrefix(token, "Bearer ")

        uuid, err := ValidateJWT(token)
        if err != nil {
            return nil, status.Error(codes.Unauthenticated, "invalid or expired token")
        }

        // we will store uuid in context
        ctx = context.WithValue(ctx, UserUUIDKey, uuid)

        return handler(ctx, req)
    }
}

Now we have to register this interceptor while creating the grpc server in the main.go file. 

1
2
3
4
// create a gRPC server object
srv := grpc.NewServer(
    grpc.UnaryInterceptor(lib.JWTInterceptor()),
)

Now if we make any call without adding the JWT token in the request metadata or add an expired or invalid JWT token the RPC calls will fail.

Note: We will pass the JWT token in the request metadata in this way: The key will be authorization and value will be Bearer <jwt token>.

Missing JWT token error in grpcui
Missing JWT token error in grpcui

Attaching user_id

We will use the context to get the uuid of the user and in the CreateTodo RPC and we will add that to the todo we are saving in the database. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
func (ts *TodoService) CreateTodo(ctx context.Context, req *pb.CreateTodoRequest) 
(*pb.CreateTodoResponse, error) {
    // get the uuid of the user from context
    uuid, ok := ctx.Value(lib.UserContextUUIDKey).(string)
    if !ok || uuid == "" {
        return nil, status.Error(codes.Unauthenticated, "unable to identify user")
    }

    // create the todo
    todo := &pb.Todo{
        Title:     strings.TrimSpace(req.GetTitle()),
        Priority:  req.GetPriority(),
        Done:      false,
        CreatedAt: timestamppb.Now(),
        UserUuid:  uuid,
    }

    if req.GetDescription() != "" {
        desc := strings.TrimSpace(req.GetDescription())
        todo.Description = &desc
    }

    // save in DB
    todo, err := ts.repo.Save(todo)
    if err != nil {
        log.Printf("error creating todo: %+v", err)
        return nil, status.Error(codes.Internal, fmt.Sprintf("failed to create Todo: %v", err))
    }

    // return the response
    return &pb.CreateTodoResponse{
        Todo: todo,
    }, nil
}

Once we do this we can see that the user_id is getting attached to the todo and is also saved in the database. 

1
2
3
4
5
6
7
8
9
{
    "title": "Implement JWT in gRPC",
    "description": "We will implement the interceptor and wire up the missing parts",
    "done": false,
    "priority": "PRIORITY_LOW",
    "created_at": "2023-02-25T13:51:06Z",
    "uuid": "b78423df-7395-4142-95b3-657aac419fb2",
    "user_uuid": "0901b0c0-7409-43a5-87b8-0aedd351b9d9"
}

Caution: Handle Null UUID

Now when we have wired up the todos with the user_id we will face a small glitch while calling the GetTodo and ListTodos. The glitch will be caused by the null values in the user_id field of the existing todo records in the database. Let’s fix that in the repository layer: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
func (r *MySQLTodoRepository) FindByID(id string) (*pb.Todo, error) {
    query := "SELECT uuid, title, description, is_done, priority, user_id, created_at FROM todos WHERE uuid = ?"

    row := r.db.QueryRow(query, id)
    todo := &pb.Todo{}

    var priority string
    var created_at time.Time
    var user_uuid sql.NullString

    if err := row.Scan(
        &todo.Uuid, &todo.Title, &todo.Description, &todo.Done,
        &priority, &user_uuid, &created_at,
    ); err != nil {
        log.Printf("error finding todo: %+v", err)
        return nil, err
    }

    // attach created_at
    todo.CreatedAt = timestamppb.New(created_at)

    // attach user_uuid
    if user_uuid.Valid {
        todo.UserUuid = user_uuid.String
    }

    // attach priority
    switch priority {
    case "low":
        todo.Priority = pb.Todo_PRIORITY_LOW
    case "medium":
        todo.Priority = pb.Todo_PRIORITY_MEDIUM
    case "high":
        todo.Priority = pb.Todo_PRIORITY_HIGH
    default:
        todo.Priority = pb.Todo_PRIORITY_LOW
    }

    return todo, nil
}

func (r *MySQLTodoRepository) FindAll(last_created_at time.Time, size int32) ([]*pb.Todo, error) {
    query := "SELECT uuid, title, description, is_done, priority, user_id, created_at FROM todos"

    if !last_created_at.IsZero() {
        query += " WHERE created_at < '" + last_created_at.Format("2006-01-02 15:04:05") + "'"
    }

    query += " ORDER BY created_at DESC"

    if size > 0 {
        query += " LIMIT " + strconv.Itoa(int(size))
    }

    rows, err := r.db.Query(query)
    if err != nil {
        log.Printf("error finding todos: %+v", err)
        return nil, err
    }
    defer rows.Close()

    var todos []*pb.Todo
    for rows.Next() {
        // holders
        var todo pb.Todo
        var priority string
        var created_at time.Time
        var user_uuid sql.NullString

        if err := rows.Scan(
            &todo.Uuid, &todo.Title, &todo.Description, &todo.Done,
            &priority, &user_uuid, &created_at,
        ); err != nil {
            return nil, err
        }

        // attach created_at
        todo.CreatedAt = timestamppb.New(created_at)

        // attach user_uuid
        if user_uuid.Valid {
            todo.UserUuid = user_uuid.String
        }

        // attach priority
        switch priority {
        case "low":
            todo.Priority = pb.Todo_PRIORITY_LOW
        case "medium":
            todo.Priority = pb.Todo_PRIORITY_MEDIUM
        case "high":
            todo.Priority = pb.Todo_PRIORITY_HIGH
        default:
            todo.Priority = pb.Todo_PRIORITY_LOW
        }

        todos = append(todos, &todo)
    }

    return todos, nil
}

Restrict Manipulations

Finally while updating or deleting the todo, we will validate if the caller of the RPC is actually the owner of the todo or not. Accordingly we will respond to the requests.

This is actually the concept of Authorization.

Authentication validates if the system can identify the caller but authorization validates whether the caller can perform certain actions on a resource.

Let’s make the required changes in the service layer. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

func (ts *TodoService) UpdateTodo(ctx context.Context, req *pb.UpdateTodoRequest) (*pb.UpdateTodoResponse, error) {
    if req.GetUuid() == "" {
        return nil, status.Error(codes.InvalidArgument, "ID should be positive")
    }

    // get the uuid of the user from context
    uuid, ok := ctx.Value(lib.UserContextUUIDKey).(string)
    if !ok || uuid == "" {
        return nil, status.Error(codes.Unauthenticated, "unable to identify user")
    }

    // get todo by id
    val, err := ts.repo.FindByID(req.GetUuid())
    if err != nil {
        log.Printf("error updating todo: %+v", err)
        return nil, status.Error(codes.Internal, fmt.Sprintf("failed to get Todo: %v", err))
    }

    if val.GetUserUuid() != uuid {
        return nil, status.Error(codes.PermissionDenied, "you don't have permission to update this todo")
    }

    // update the values from req
    val.Title = req.GetTitle()
    val.Done = req.GetDone()
    val.Uuid = req.GetUuid()

    if req.GetDescription() != "" {
        desc := req.GetDescription()
        val.Description = &desc
    }

    // update the record
    err = ts.repo.UpdateTodo(val)
    if err != nil {
        log.Printf("error updating todo: %+v", err)
        return nil, status.Error(codes.Internal, fmt.Sprintf("failed to update Todo: %v", err))
    }

    // return response
    return &pb.UpdateTodoResponse{
        Todo: val,
    }, nil
}

func (ts *TodoService) DeleteTodo(ctx context.Context, req *pb.DeleteTodoRequest) 
(*emptypb.Empty, error) {
    if req.GetUuid() == "" {
        return nil, status.Error(codes.InvalidArgument, "ID should be positive")
    }

    // get the uuid of the user from context
    uuid, ok := ctx.Value(lib.UserContextUUIDKey).(string)
    if !ok || uuid == "" {
        return nil, status.Error(codes.Unauthenticated, "unable to identify user")
    }

    // get todo by id
    val, err := ts.repo.FindByID(req.GetUuid())
    if err != nil {
        log.Printf("error updating todo: %+v", err)
        return nil, status.Error(codes.Internal, fmt.Sprintf("failed to get Todo: %v", err))
    }
    if val.GetUserUuid() != uuid {
        return nil, status.Error(codes.PermissionDenied, "you don't have permission to update this todo")
    }

    err = ts.repo.Delete(req.GetUuid())
    if err != nil {
        log.Printf("error deleting todo: %+v", err)
        return nil, status.Error(codes.Internal, fmt.Sprintf("failed to delete Todo: %v", err))
    }

    // return response
    return &emptypb.Empty{}, nil
}

Further Improvements

Now that we have JWT code implemented and we are providing the JWT token to the user, we have to fix a few more loose ends. Let me list down them or you might already know.

  • The user email and username should be unique.
  • Using migrations to track the database changes.
  • Using timeouts from the client and handling them in the server

Fixing MySQL Tables

In the users table we have to make the username and the email unique so that we do not end up having multiple rows for same user. We can use the alter command to make the field value unique. 

1
2
3
ALTER TABLE users
  ADD CONSTRAINT uqx_users_email UNIQUE (email),
  ADD CONSTRAINT uqx_users_username UNIQUE (username);

Migrations

Now we can see that there are series of changes for the database as well and we must keep a track of them so that we can replicate them to multiple environments. Not only that we can even revert them in case we feel that a particular database change was not fruitful.

We carry out such tracked database changes va migrations. We will look into that in future posts.

Conclusion

In this post, we took our gRPC application a step closer to production readiness by adding pagination, authentication, and authorization. We implemented cursor-based pagination to handle large datasets efficiently, secured our RPCs using JWT, and leveraged gRPC interceptors to keep authentication logic out of our core services.

By enforcing ownership checks at the service layer, we ensured that users can only access and manipulate their own data—clearly separating authentication from authorization.

These patterns are widely used in real-world gRPC systems and form a strong foundation for building secure, scalable backend services. In the upcoming posts, we will continue refining this application by focusing on operational and scalability aspects, including database migrations, request deadlines, and better error handling strategies.

Stay healthy, stay blessed!