Efficient way to display fast changing images in Blazor Server/Hybrid

I'm considering using Blazor Hybrid to rewrite my app. So far, I'm loving the productivity, but one of the core features is image processing.

To check whether the Blazor Hybrid is a valid option, I created a sample WPF app, added Blazor and I noticed that just binding to a source of <img> tag is extremely slow.

Here is a component I created:

@using System.Reactive.Subjects
@using System.Reactive
@using System.Reactive.Linq
@using SkiaSharp
@using MudBlazor


<h3>FollowCursor</h3>
<MudCheckBox @bind-Checked="@hideImage">Hide image with opacity</MudCheckBox>
<MudCheckBox @bind-Checked="@useVisibillityHidden">Visibillity hidden</MudCheckBox>
<MudCheckBox @bind-Checked="@useImageSource">Use image source</MudCheckBox>
<MudCheckBox @bind-Checked="@disableStateHasChangedOnMove">Disable StateHasChanged on move</MudCheckBox>

<p>Mouse position: @MousePosition</p>
<p>Image size: @ImageSize</p>
<div style="width: 100%; height: 500px;border: solid green 1px" @ref="Img" @onmousemove="disableStateHasChangedOnMove ? EventUtil.AsNonRenderingEventHandler<MouseEventArgs>(onMouseMove) : onMouseMove">
    <img  src="@ImageSource" style="height: 100%; width: 100%; margin: auto; border: solid red 1px;opacity: @(hideImage ? 0 : 1); visibility: @(useVisibillityHidden ? "hidden" : "visible")" 
         />
</div>


@code {

    public ElementReference Img { get; set; }

    public string? ImageSource
    {
        get => useImageSource ?  _imageSource : null;
        set => _imageSource = value;
    }

    public SKPoint MousePosition { get; set; }
    public SKSize ImageSize { get; set; }

    Subject<Unit> _mouseMove = new();
    private string? _imageSource;
    
    bool useImageSource = true;
    bool hideImage = false;
    bool useVisibillityHidden = false;
    bool disableStateHasChangedOnMove = true;

    protected override async Task OnInitializedAsync()
    {
        StateHasChanged();
        _mouseMove
            // .Sample(TimeSpan.FromMilliseconds(1))
            .Do(async _ =>
        {
            var drawMousePosition = DrawMousePosition();
            await InvokeAsync(() =>
            {
                ImageSource = drawMousePosition;
                if (disableStateHasChangedOnMove)
                {
                    StateHasChanged();
                }
                
            });
        }).Subscribe();
    }

    private string DrawMousePosition()
    {
        var bmp = new SKBitmap((int)ImageSize.Width, (int)ImageSize.Height);
        using var canvas = new SKCanvas(bmp);
        canvas.Clear(SKColors.White);

        canvas.DrawCircle(MousePosition.X, MousePosition.Y, 10, new SKPaint
        {
            Color = SKColors.Red,
            Style = SKPaintStyle.Fill
        });

        canvas.Save();

        return ToBase64Image(bmp);
    }

    private async Task onMouseMove(MouseEventArgs e)
    {
        MousePosition = new SKPoint((float)e.OffsetX, (float)e.OffsetY);
        _mouseMove.OnNext(Unit.Default);
    }


    protected override async Task OnAfterRenderAsync(bool firstRender)
    {
        await base.OnAfterRenderAsync(firstRender);

        var rect = await Img.MudGetBoundingClientRectAsync();

        ImageSize = new SKSize((float)rect.Width, (float)rect.Height);
    }
    
    public static string ToBase64Image( SKBitmap bmp)
    {
        using var image = SKImage.FromBitmap(bmp);
        using var data = image.Encode(SKEncodedImageFormat.Png, 100);

        return "data:image/png;base64," + Convert.ToBase64String(data.ToArray());
    }
}

Its job is to draw a circle under your cursor. This is how I achieved various drawing tools in the WPF app I am trying to replace - for example when you are drawing a line, I generate a preview image every time your mouse moves and refresh it. When LMB is released, latest preview replaces the edited image.

In WPF or AvaloniaUI this approach works extremely well.

In Blazor, it is really slow. I believe that the reason is converting base64 image to a bitmap in the BlazorWebView is the bottleneck, you can verify with the checkboxes. Also, I know its a good idea to silence the autotriggering of StateHasChanged on mouse move, but the result still isn't great.

Is there a better way to do this? I know that there is a special high performance view component for SkiaSharp, but it works only with Blazor WA - it is based on reusing memory between renderer and the bitmap itself. Not possible in Hybrid and server. There is an option of creating this in Blazor WA and publishing it as a custom component and embeding it in the Hybrid app, but it seems like a weird workaround.

As an alternative, I know that I can create the image editing part in the host tech for Hybrid (like maui to get multiplatform support), and show it on top of when needed, but maybe there is a way to do this in Blazor.

I found some canvas wrappers, but they still require to create the img tag with bound source, and this is the slowest part.

Any suggestions? Maybe the whole idea behind how to draw the preview is wrong?

Recommended topics

Hot tags