C# gRPC file streaming, original file smaller than the streamed one

Question

I am having some problems with setting up a request-stream type gRPC architecture. The code below is just for testing purposes and it is missing various validation checks, but the main issue is that the original file is always smaller than the received one.

Could the cause here be encoding? It doesn't matter what the file type is, the end result is always that the file sizes are different.

Protobuf inteface:

syntax = "proto3";
package FileTransfer;
option csharp_namespace = "FileTransferProto";

service FileTransferService {  
    rpc DownloadFile(FileRequest) returns (stream ChunkMsg);
}  

message ChunkMsg {
    string FileName = 1;
    int64 FileSize = 2;
    bytes Chunk = 3;
}

message FileRequest {
    string FilePath = 1;
}

Server side (sending):

    public override async Task DownloadFile(FileRequest request, IServerStreamWriter<ChunkMsg> responseStream, ServerCallContext context)
    {
        string filePath = request.FilePath;

        if (!File.Exists(filePath)) { return; }

        FileInfo fileInfo = new FileInfo(filePath);

        ChunkMsg chunk = new ChunkMsg();
        chunk.FileName = Path.GetFileName(filePath);
        chunk.FileSize = fileInfo.Length;

        int fileChunkSize = 64 * 1024;

        byte[] fileByteArray = File.ReadAllBytes(filePath);
        byte[] fileChunk = new byte[fileChunkSize];
        int fileOffset = 0;

        while (fileOffset < fileByteArray.Length && !context.CancellationToken.IsCancellationRequested)
        {
            int length = Math.Min(fileChunkSize, fileByteArray.Length - fileOffset);
            Buffer.BlockCopy(fileByteArray, fileOffset, fileChunk, 0, length);
            fileOffset += length;
            ByteString byteString = ByteString.CopyFrom(fileChunk);

            chunk.Chunk = byteString;
            await responseStream.WriteAsync(chunk).ConfigureAwait(false);
        }            
    }

Client side (receiving):

    public static async Task GetFile(string filePath)
    {
        var channel = Grpc.Net.Client.GrpcChannel.ForAddress("https://localhost:5001/", new GrpcChannelOptions
        {
            MaxReceiveMessageSize = 5 * 1024 * 1024, // 5 MB
            MaxSendMessageSize = 5 * 1024 * 1024, // 5 MB
        });

        var client = new FileTransferProto.FileTransferService.FileTransferServiceClient(channel);

        var request = new FileRequest { FilePath = filePath };
        string tempFileName = $"temp_{DateTime.UtcNow.ToString("yyyyMMdd_HHmmss")}.tmp";
        string finalFileName = tempFileName;

        using (var call = client.DownloadFile(request))
        {
            await using (Stream fs = File.OpenWrite(tempFileName))
            {
                await foreach (ChunkMsg chunkMsg in call.ResponseStream.ReadAllAsync().ConfigureAwait(false))
                {
                    Int64 totalSize = chunkMsg.FileSize;
                    string tempFinalFilePath = chunkMsg.FileName;

                    if (!string.IsNullOrEmpty(tempFinalFilePath))
                    {
                        finalFileName = chunkMsg.FileName;
                    }

                    fs.Write(chunkMsg.Chunk.ToByteArray());
                }
            }
        }

        if (finalFileName != tempFileName)
        {
            File.Move(tempFileName, finalFileName);
        }
    }

Yarbrough

Hi there - can I clarify something? You've tagged this protobuf-net, but the code shown doesn't look like protobuf-net / protobuf-net.Grpc ; can I just check: you're using the vanilla Google API here, yes? – Septuple

In your send code, it you write out length to the console (or whatever) each time, and write out all the received lengths each time: do they match? – Septuple

I also notice that the chunk you send doesn't actually depend on length, which sounds very bad. Is there an overload of the ByteString constructor that takes a length? – Septuple

Answer 1

To add to Marc's answer, I feel like you can simplify your code a little bit.

using var fs = File.Open(filePath, System.IO.FileMode.Open);
int bytesRead;
var buffer = new byte[fileChunkSize];
while ((bytesRead = await fs.ReadAsync(buffer)) > 0)
{
     await call.RequestStream.WriteAsync(new ChunkMsg
     {
          // Here the correct number of bytes must be sent which is starting from
          // index 0 up to the number of read bytes from the file stream.
          // If you solely pass 'buffer' here, the same bug would be present.
          Chunk = ByteString.CopyFrom(buffer[0..bytesRead]),
     });
}

I've used the array range operator from C# 8.0 which makes this cleaner or you can also use the overload of ByteString.CopyFrom which takes in an offset and count of how many bytes to include.

Fullerton

Answer 2

In your write loop, the chunk you actually send is for the oversized buffer, not accounting for length. This means that the last segment includes some garbage and is oversized. The received payload will be oversized by this same amount. So: make sure you account for length when constructing the chunk to send.

Septuple

Answer 3

I tested the code and modified it to transfer the correct size.

The complete code is available at the following URL: https://github.com/lisa3907/grpc.fileTransfer

server-side-code

 while (_offset < _file_bytes.Length)
 {
    if (context.CancellationToken.IsCancellationRequested)
    break;
 
    var _length = Math.Min(_chunk_size, _file_bytes.Length - _offset);
    Buffer.BlockCopy(_file_bytes, _offset, _file_chunk, 0, _length);
 
    _offset += _length;
 
    _chunk.ChunkSize = _length;
    _chunk.Chunk = ByteString.CopyFrom(_file_chunk);
 
    await responseStream.WriteAsync(_chunk).ConfigureAwait(false);
 }

client-side-code

await foreach (var _chunk in _call.ResponseStream.ReadAllAsync().ConfigureAwait(false))
{
    var _total_size = _chunk.FileSize;

    if (!String.IsNullOrEmpty(_chunk.FileName))
    {
       _final_file = _chunk.FileName;
    }

    if (_chunk.Chunk.Length == _chunk.ChunkSize)
       _fs.Write(_chunk.Chunk.ToByteArray());
    else
    {
       _fs.Write(_chunk.Chunk.ToByteArray(), 0, _chunk.ChunkSize);
       Console.WriteLine($"final chunk size: {_chunk.ChunkSize}");
    }
}

Intercellular