C
C#ā€¢2y ago
leviathan_0

Rendering and updating a colormap

I'm developing a library related to real-time image generation. To test it, I'm trying to make an app that can convert a Color[,] into an image and display it, updating at 60 fps (Color is a class with 3 float fields). Ideally, I'd want my code to be like this:
DynamicImage dynImg = new DynamicImage();
every 1/60 seconds do{
Color[,] colormap = dynImg.getImgAtTime(time);
imageDisplay.setImage(colormap.convertToImage());
}
DynamicImage dynImg = new DynamicImage();
every 1/60 seconds do{
Color[,] colormap = dynImg.getImgAtTime(time);
imageDisplay.setImage(colormap.convertToImage());
}
where imageDisplay would probably be a PictureBox in a WPF app (with which I have no experience at all). (This is of course pseudocode) Any help is greatly appreciated.
75 Replies
leviathan_0
leviathan_0ā€¢2y ago
@GUI
Sergio
Sergioā€¢2y ago
There's multiple ways you could go about this: - Use a WriteableBitmap - Use Win2D and a canvas control/canvas image source, and draw manually - Use ComputeSharp - etc. Also: I would really avoid allocating an array for each frame, that's uber expensive
leviathan_0
leviathan_0ā€¢2y ago
If it isn't too bothersome, could you please explain how exactly should I do it using a WriteableBitmap?
SuperBrain
SuperBrainā€¢2y ago
where imageDisplay would probably be a PictureBox in a Windows Forms
You've chosen a bad GUI framework and the worst possible control for this... If you want performance, at the very least, you should use .NET 6, rather than .NET Framework 4.x
leviathan_0
leviathan_0ā€¢2y ago
As I've stated, I have no experience whatsoever with Windows Forms. I am using .NET 6, I've never mentioned .NET Framework
SuperBrain
SuperBrainā€¢2y ago
Ok, but generally, the underlying GUI framework you want to use is going to define all the other capabilities available to you. As much as I hate UWP, it still is more performant than anything else. Personally, I would suggest using WPF, but this will depend on availability of features Sergio mentioned before Technically speaking, if you know what you're doing, there are ways to make WinForms apps very fast, but most of the times that requires using low-level APIs.
leviathan_0
leviathan_0ā€¢2y ago
I'm using WPF, which I thought wasthe same as Windows Forms, but it seems I'm mistaken
SuperBrain
SuperBrainā€¢2y ago
No, WPF is quite different and it's way more capable than WinForms. But WPF has it's own limitations. Something that wasn't yet mentioned here is SkiaSharp. IIRC, that lib has some features that may interest you.
leviathan_0
leviathan_0ā€¢2y ago
This app is only to test a library that I'm developing, so it doesn't need to be anything fancy, all I want to do is to 'display' a 2d array of colours that changes over time
SuperBrain
SuperBrainā€¢2y ago
Then you probably want to try using ComputeShart or WritableBitmap
leviathan_0
leviathan_0ā€¢2y ago
Well, that comes back to what I asked, WriteableBitmap seems to be the best option for my use case, but I don't know how to use it and documentation isn't clear either.
SuperBrain
SuperBrainā€¢2y ago
I think @Klarth used it and may be able to help with that.
leviathan_0
leviathan_0ā€¢2y ago
Then I'll wait for @Klarth to come online
Klarth
Klarthā€¢2y ago
Check out https://github.com/stevemonaco/WpfRealTimeBitmap I also have it ported to Avalonia: https://github.com/stevemonaco/AvaloniaDemos/tree/master/RealTimeBitmapAdapter It's very close to what you want and the approach should map pretty well to UWP, too, if you want to do that. I could have skipped the indirection of creating a separate color map (with pixel values [0, 1]) and directly written randomized grayscale to the WriteableBitmap, but that didn't fit my scenario.
leviathan_0
leviathan_0ā€¢2y ago
It seems that my project can't access System.Windows.Media, it says 'missing assembly reference'. How can I add a reference to PresentationCore.dll?
Klarth
Klarthā€¢2y ago
This is a WPF app, not a WinForms app. I strongly suggest you not use WinForms, especially if this is basically all the functionality you need.
leviathan_0
leviathan_0ā€¢2y ago
Ah, that confusion has already been sorted out, my project is indeed a WPF app
Klarth
Klarthā€¢2y ago
There's no reason why you should have to add a reference to that if you've created a WPF .NET 6 app. Are you sure you didn't accidentally use the WPF .NET Framework project template?
leviathan_0
leviathan_0ā€¢2y ago
According to the properties window, I'm using .NET 6, not .NET Framework
Klarth
Klarthā€¢2y ago
I'm not sure how that could be the case. šŸ¤” Can you paste your .csproj?
leviathan_0
leviathan_0ā€¢2y ago
Wait.. What!!?? I clearly remember choosing WPF, and yet the properties window looks like this:
leviathan_0
leviathan_0ā€¢2y ago
Is it enough to just swap them, or should I create a new project?
Klarth
Klarthā€¢2y ago
If you don't have XAML files (eg. App.xaml, etc) in your project, then recreate because you didn't create a WPF project. Otherwise, it might be ok to swap. I'd probably just recreate though to save potential headaches.
leviathan_0
leviathan_0ā€¢2y ago
I see, I'll recreate it then, no XAML on my end... So... uh... how does WPF work?????
Klarth
Klarthā€¢2y ago
$rulesofwpf
MODiX
MODiXā€¢2y ago
Rules of WPF:

āŒ Avoid the WPF Designer to eliminate a category of confusing bugs
āŒ Don't rely on Margin as the primary tool for layouts
āŒ Avoid writing UserControls or subclassing to extend a default control -- use Behaviors instead (Microsoft.Xaml.Behaviors.Wpf)

āœ… Write XAML by hand and autoformat with "Ctrl K,D" or XAML Styler
āœ… Rely upon XAML Hot Reload to design your app's UI at runtime
āœ… Use layout controls (Grid, DockPanel, etc) to support proper resizing
āœ… Use data binding to eliminate glue code and state synchronization issues
āœ… Use collection controls and DataTemplate to dynamically create lists of controls
āœ… Learn MVVM to create maintainable apps
āœ… Use the Dispatcher to update controls from non-UI threads
āœ… WPF's default controls can be easily modernized via $wpfuilibs
āœ… Include relevant XAML, code-behind, and ViewModel code for questions when possible
Rules of WPF:

āŒ Avoid the WPF Designer to eliminate a category of confusing bugs
āŒ Don't rely on Margin as the primary tool for layouts
āŒ Avoid writing UserControls or subclassing to extend a default control -- use Behaviors instead (Microsoft.Xaml.Behaviors.Wpf)

āœ… Write XAML by hand and autoformat with "Ctrl K,D" or XAML Styler
āœ… Rely upon XAML Hot Reload to design your app's UI at runtime
āœ… Use layout controls (Grid, DockPanel, etc) to support proper resizing
āœ… Use data binding to eliminate glue code and state synchronization issues
āœ… Use collection controls and DataTemplate to dynamically create lists of controls
āœ… Learn MVVM to create maintainable apps
āœ… Use the Dispatcher to update controls from non-UI threads
āœ… WPF's default controls can be easily modernized via $wpfuilibs
āœ… Include relevant XAML, code-behind, and ViewModel code for questions when possible
Klarth
Klarthā€¢2y ago
You write XAML for layout / styling. You can use code-behind, too, but it should be limited in scope. The designer sucks compared to WinForms. I disable it and suggest that newcomers disable too, after a few days.
leviathan_0
leviathan_0ā€¢2y ago
So how can I replicate this kind of design in XAML? (it's an image and two text boxes)
SuperBrain
SuperBrainā€¢2y ago
<Grid>
<Grid.RowDefinitions>
<RowDefinition />
<RowDefinition />
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition />
<ColumnDefinition />
</Grid.ColumnDefinitions>
</Grid>
<Grid>
<Grid.RowDefinitions>
<RowDefinition />
<RowDefinition />
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition />
<ColumnDefinition />
</Grid.ColumnDefinitions>
</Grid>
Start with this šŸ‘€
Klarth
Klarthā€¢2y ago
Klarth
Klarthā€¢2y ago
leviathan_0
leviathan_0ā€¢2y ago
Thanks a lot for taking the time to do this! I'll try to understand what you're doing, it shouldn't be to hard.
Klarth
Klarthā€¢2y ago
The layout philosophy is a lot different from WinForms, if you were used to that.
leviathan_0
leviathan_0ā€¢2y ago
Good thing that I wasn't then
Klarth
Klarthā€¢2y ago
WPF uses layout controls with their own sizing characteristics instead of the anchoring system that WinForms uses. Basically, this is a 3x3 grid with proportional sizing.
SuperBrain
SuperBrainā€¢2y ago
You should be able to learn a lot by going through this site https://wpf-tutorial.com/
leviathan_0
leviathan_0ā€¢2y ago
The code is actually quite straightforward, except for one part: What are the asterisks after some of the sizes in the Row/ColumnDefinitions? Thanks, I'll check it out
Klarth
Klarthā€¢2y ago
If they sum to 100, like in this case, you can think of them as percentages of the remaining proportional space. If you had two cells only, one with 2* and the other with *, then the sum is 3. So the first would be 66% and the second would be 33%.
leviathan_0
leviathan_0ā€¢2y ago
So they are fractional relative units, I see. Nothing too out there, I've seen them before. In this, you use a Binding for the image source. How do those work?
Klarth
Klarthā€¢2y ago
That's a lot to go through. Roughly, bindings associate a property on the DataContext with a dependency property on a control. There are numerous configurations ranging from indirection and how/if the updates occur.
leviathan_0
leviathan_0ā€¢2y ago
What's the purpose of a BitmapAdapter?
Klarth
Klarthā€¢2y ago
Self-contained type to manage the copying from the native image concept ColorScaleImage to the WPF image concept WriteableBitmap. Well, that's the specific ColorScaleBitmapAdapter. BitmapAdapter is a general abstraction. Keep in mind, these are patterns for a medium-sized app that is very generalizable. So it might not make sense to you when you're trying to scaffold a new, tiny project. I could remove all of the abstractions and just do everything in one code-behind file, but that doesn't scale with project complexity.
leviathan_0
leviathan_0ā€¢2y ago
if (!Bitmap!.TryLock(new System.Windows.Duration(TimeSpan.FromMilliseconds(500))))
if (!Bitmap!.TryLock(new System.Windows.Duration(TimeSpan.FromMilliseconds(500))))
What is the !. operator in this line?
Klarth
Klarthā€¢2y ago
$handlenullable
MODiX
MODiXā€¢2y ago
If you're getting warnings related to nullables, here's how you can handle it. 1. Forward Just let it go šŸŽ¶
string? expr = Console.ReadLine();
string[]? words = expr?.Split();
string? expr = Console.ReadLine();
string[]? words = expr?.Split();
2. Suppress By adding !, like here:
string expr = Console.ReadLine()!;
string expr = Console.ReadLine()!;
3. Handle null
string expr = Console.ReadLine() ?? throw new("Oh no! Got a null");
string expr = Console.ReadLine() ?? throw new("Oh no! Got a null");
or
if (Console.ReadLine() is { } validThing)
{
...
}
if (Console.ReadLine() is { } validThing)
{
...
}
Klarth
Klarthā€¢2y ago
Part of null-forgiving operator.
leviathan_0
leviathan_0ā€¢2y ago
In this case, shouldn't it be
if (!Bitmap?.TryLock(new System.Windows.Duration(TimeSpan.FromMilliseconds(500))))
if (!Bitmap?.TryLock(new System.Windows.Duration(TimeSpan.FromMilliseconds(500))))
Using the null-forgiving member-access operator?
Klarth
Klarthā€¢2y ago
Probably. It's a small demo that was updated from an older C#, so I don't care that much.
leviathan_0
leviathan_0ā€¢2y ago
Fair enough
leviathan_0
leviathan_0ā€¢2y ago
I'm not sure how I can convert my Image class to a WriteableBitmap... Could you please check the code at https://github.com/levi-gomes/RIRE and give me some clues?
GitHub
GitHub - levi-gomes/RIRE
Contribute to levi-gomes/RIRE development by creating an account on GitHub.
leviathan_0
leviathan_0ā€¢2y ago
I've of course looked at ColorScaleBitmapAdapter.Render, but I don't quite understand what's going on...
Klarth
Klarthā€¢2y ago
Before that, what is https://github.com/levi-gomes/RIRE/blob/master/RIRE/Color.cs ? Is this just one pixel? Or is it many?
leviathan_0
leviathan_0ā€¢2y ago
It's a colour - so I guess it is a single pixel.
Klarth
Klarthā€¢2y ago
Ok, so this is going to absolutely tank the performance. First, it should be a struct and not a class. As-is, your image array is an array of pointers, so there's a lot of extra allocations + indirection. I'm guessing each color channel is normalized between 0-1?
leviathan_0
leviathan_0ā€¢2y ago
Yes, it is
Klarth
Klarthā€¢2y ago
I'd start with something more like:
public readonly struct Color
{
public float R => _r;
public float G => _g;
public float B => _b;

private float _r;
private float _g;
private float _b;

public Color(float r = 0, float g = 0, float b = 0)
{
_r = r;
_g = g;
_b = b;
}
}
public readonly struct Color
{
public float R => _r;
public float G => _g;
public float B => _b;

private float _r;
private float _g;
private float _b;

public Color(float r = 0, float g = 0, float b = 0)
{
_r = r;
_g = g;
_b = b;
}
}
You can remove the readonly and make it mutable if you want...but that will solve most of the perf issues that will happen.
leviathan_0
leviathan_0ā€¢2y ago
There's no harm in keeping the indexer, is there?
Klarth
Klarthā€¢2y ago
There is...you don't want to store an array...because of even more allocation+indirection costs. You could do a fixed array, but I'd recommend against it. It's not sensible to index a single pixel, IMO.
leviathan_0
leviathan_0ā€¢2y ago
I've refactored it as such:
public struct Color
{
private float _r;
private float _g;
private float _b;

public float r => _r;
public float g => _g;
public float b => _b;

public float this[int index] {
get => index switch {
0 => _r,
1 => _g,
2 => _b,
_ => throw new ArgumentOutOfRangeException(nameof(index))
};
set => {
switch(value) {
case 0:
_r = value; break;
case 1:
_g = value; break;
case 2:
_b = value; break;
default:
throw new ArgumentOutOfRangeException(nameof(value));
}
}
}

public Color(float r = 0, float g = 0, float b = 0) {
this._r = r;
this._g = g;
this._b = b;
}
}
public struct Color
{
private float _r;
private float _g;
private float _b;

public float r => _r;
public float g => _g;
public float b => _b;

public float this[int index] {
get => index switch {
0 => _r,
1 => _g,
2 => _b,
_ => throw new ArgumentOutOfRangeException(nameof(index))
};
set => {
switch(value) {
case 0:
_r = value; break;
case 1:
_g = value; break;
case 2:
_b = value; break;
default:
throw new ArgumentOutOfRangeException(nameof(value));
}
}
}

public Color(float r = 0, float g = 0, float b = 0) {
this._r = r;
this._g = g;
this._b = b;
}
}
Is this fine? Or should I completely get rid of that indexer?
Klarth
Klarthā€¢2y ago
I would get rid of it. You're going to have issues with value-type copy semantics at some point, I think. struct should typically be immutable because of that.
leviathan_0
leviathan_0ā€¢2y ago
Well, I don't think I'll really need it, so I'll take it out. As for making it readonly, would it still allow for something like some_color.r+=0.1f?
Klarth
Klarthā€¢2y ago
No...because it wouldn't be readonly if you could modify it.
leviathan_0
leviathan_0ā€¢2y ago
Then I'll try to get away with it being mutable, if I do get issues I'll change to immutable
Klarth
Klarthā€¢2y ago
var newColor = new Color(some_color.R + 0.1f, some_color.G, some_color.B); You could add some methods to clean it up.
leviathan_0
leviathan_0ā€¢2y ago
Alright, I'll make it readonly and deal with it. That can be a future me problem.
Klarth
Klarthā€¢2y ago
https://paste.mod.gg/cxhounnbbfcs/0 So your implementations would look something like that. It's more efficient to use Span<T> than your multidimensional array, but your image type doesn't support that. Some of the perf gap will be closed in .NET 7 with MD arrays.
leviathan_0
leviathan_0ā€¢2y ago
So I would add this to MainWindow and invoke it from RenderFrame, am I right?
Klarth
Klarthā€¢2y ago
How you integrate it depends on you. That's what I did, but I didn't want to use MVVM in the WPF version.
leviathan_0
leviathan_0ā€¢2y ago
Sorry, what is MVVM?
Klarth
Klarthā€¢2y ago
The Avalonia version is a bit more MVVM. Model-View-ViewModel architecture. It's a way of separating concerns to make individual components more simplified at the expense of having many more components.
leviathan_0
leviathan_0ā€¢2y ago
Unfortunately I have to go now. I'll try to integrate it at a later date (probably tomorrow). I'll let you know if I manage to do it (or if I have any more questions...). I can't thank you enough for your help!
Sergio
Sergioā€¢2y ago
Compute... Shart? harold
Sergio
Sergioā€¢2y ago
The docs show an example. It's a bit awkward as the API isn't the best, but it does the trick. Essentially you just have a grid of pixels, so you can lock the bitmap, update them, unlock the bitmap, every frame: https://learn.microsoft.com/en-us/dotnet/api/system.windows.media.imaging.writeablebitmap
WriteableBitmap Class (System.Windows.Media.Imaging)
Provides a BitmapSource that can be written to and updated.
Sergio
Sergioā€¢2y ago
Also what framework are you using?
leviathan_0
leviathan_0ā€¢2y ago
I've got it working!