Nullable types - TimestampValue and DurationValue

[zs-dima]

It is nice to have Int32Value, StringValue and etc nullable type for the int32, string and etc non-nullable types.

But TimestampValue and DurationValue nullable types are missing - so it confusing many developers:
grpc/grpc-dart#502

Could I kindly ask you to implement TimestampValue and DurationValue nullable types ?

[mattwelke]

These types actually already have support for nullability. In proto3, any type that is a message has built in support for nullability. Because Timestamp and Duration are messages, it works out of the box.

Example: https://github.com/mattwelke/dart-proto-example

Here's my test message, based on the official Dart getting started guide:

syntax = "proto3";
package tutorial;

import "google/protobuf/timestamp.proto";
import "google/protobuf/wrappers.proto";

message Person {
  string                      first_name      = 1;
  google.protobuf.StringValue last_name       = 2;
  google.protobuf.Timestamp   graduation_date = 3;
}

I use a string and a StringValue to demonstrate the difference between a string, which is a primitive and does not support nullability, and a StringValue which because it's a message (a wrapper message) does support it.

In the main.dart file:

import 'google/protobuf/timestamp.pb.dart';
import 'google/protobuf/wrappers.pb.dart';
import 'person.pb.dart';

void main() {
  final john = new Person(firstName: "John");
  print("John has last name: ${john.hasLastName()}");
  print("John is graduate: ${john.hasGraduationDate()}\n");

  final jane =
      new Person(firstName: "Jane", lastName: new StringValue(value: "Doe"));
  print("Jane has last name: ${jane.hasLastName()} (${jane.lastName.value})");
  print("Jane is graduate: ${jane.hasGraduationDate()}\n");

  final jim = new Person(
      firstName: "Jim",
      lastName: new StringValue(value: "Doe"),
      graduationDate: new Timestamp());
  print("Jim has last name: ${jim.hasLastName()} (${jim.lastName.value})");
  print(
      "Jim has graduated: ${jim.hasGraduationDate()} (${jim.graduationDate.seconds})\n");
}

When you run it with dart run lib/src/main.dart, this is the output:

John has last name: false
John is graduate: false

Jane has last name: true (Doe)
Jane is graduate: false

Jim has last name: true (Doe)
Jim has graduated: true (0)

Note that despite the lack of a "TimestampValue" wrapper type existing, you still get the hasGraduationDate instance method generated. All message fields will get either these "has" functions generated (Java is another example I can think of) or you can compare the field with the language's "null" value (Go and C# come to mind). It's up the implementer of the language's Protobuf plugin. In this case, the implementers of the Dart plugin went with the "has method" approach.

This is why Google only created wrapper messages for double, float, int64, uint64, int32, uint32, bool, string, and bytes. They're Protobuf primitives, so it's impossible for any implementation to generate either the "has methods" or the comparisons to the null value. So Google stepped in to make the wrapper types so that people wouldn't have to. This is nice because now a StringValue in one project is the same type as a StringValue in another project, even if they're from different parties. That saves the parties from having to write glue code if we wanted to assign the value of one field in one message to another field in another message.

A bit off topic, but note that since release 3.15, the "optional" keyword came back, and you can use that for nullable primitives without having to use the wrapper type, as long as the plugin you're using also has support for it in proto3. See the accepted answer at https://stackoverflow.com/questions/42622015/how-to-define-an-optional-field-in-protobuf-3?rq=1.

[zs-dima]

@mattwelke thanks a lot for reply and looking into
as you could see Dart command reply to the issue grpc/grpc-dart#502

you could not create Person in this way currently:

  final jim = new Person(
      firstName: "Jim",
      lastName: new StringValue(value: "Doe"),
      graduationDate: null);

and new Duration() results Duration(seconds:0) so it is not replacement for the null value.
and the same for date new Timestamp() results 1970-01-01T00:00:00Z and it is not replacement for the null value.

However optional keyword looks promising, lets wait till they will be supported in dart and C# to close this issue.

[mattwelke]

@zs-dima In my example, to create a Person who explicitly had no graduationDate, instead of writing the following code:

  final jim = new Person(
      firstName: "Jim",
      lastName: new StringValue(value: "Doe"),
      graduationDate: null);

You would write the following code:

  final jim =
      new Person(firstName: "Jim", lastName: new StringValue(value: "Doe"));

That's why I had three Persons being created in the example. I wanted to demonstrate that just like how one can omit lastName (because it is a StringValue - a message) to indicate no last name, one can omit graduatedDate for the same purpose. You'll end up having false returned by hasGraduationDate, so you can check whether there is one or not when you use the message, the same way one would do, for example, in JavaScript with an object:

if (jim.graduationDate) {
  // they have a graduation date
}

[zs-dima]

@mattwelke thanks for reply

but you could not just skip arguments specially for data mappers
therefore you have to pass something as graduationDate/graduationDuration/etc

and it is not possible in the dart at least right now:

  final jim = new Person(
      firstName: "Jim",
      lastName: new StringValue(value: "Doe"),
      graduationDate: ???);

optional keyword is not supported in the dart right away and partly supported in other languages as I could see

[zs-dima]

@mattwelke for example you have 30 fields in some object and most of them are nullable dates or durations.

It is not possible to solve it in a way you described.

But it is main use for proto messages objects to be mapped into app businesses objects and back.

[mattwelke]

Ahh I see the problem you're getting at. I actually ran into that a few weeks ago using Protobuf in Java. I wanted to use a map method in a functional programming style, but the problem was I had to omit a message field if a certain condition in the source data was true. I got it to work by changing from an immutable style to a mutable style as I produced the new object.

I'll use JS for pseudocode here.

My original code was like:

const newData = oldData.map(o => ({
  a: o.a,
  b: someCondition ? someValue : null,
});

This is my preferred way of mapping data from one type to another. But then Protobuf ended up not producing the right data for the Google API I was sending it to. They didn't accept the empty object that was produced when I put null in there.

I ended up doing this, which the Google API accepted as valid:

const newData = oldData.map(o => {
  const newObj = {
    a: o.a,
  };
  
  if (o.someCondition) {
    newObj.b = someValue;
  }

  return newObj;
});

It's not as concise as other ways of writing a mapping function, but it works. I was able to skip setting the Protobuf message field when the source data's condition was met, causing the resulting Protobuf message to not have that field set (like my Person example where I don't set the field).

[zs-dima]

@mattwelke thanks for idea but it looks just Java specific solution and will not help with other languages.
Specially with Dart businesses objects where immutable objects are required for the app architecture

[zs-dima]

@mattwelke I took @mraleph temporary workaround for now, but it looks really ugly
grpc/grpc-dart#502
hope dart and c# will support optional keyword in a way to solve this issue at least soon

[mattwelke]

I feel like because I ran into this problem in Java, and again in Dart when preparing the example for you, that it means this is just how things have to be done in Protobuf. But I'm curious what the Protobuf devs will think if they look at this. And I wonder if the new support for optional will improve the situation.

Best of luck with your task.

[fowles]

I suspect this has already been fixed for C#, but kicking it to you to verify