HASS.Agent•7mo ago

High CPU and memory usage

Hey everyone - I'm having pretty high CPU usage (consistently 10-15%) and slow increase in memory usage on windows (starts at a few hundred MBs and slowly climbs to GBs). I submitted a issue on GitHub (#64) but wanted to see if anyone else is experiencing this. Running 2.1.0-beta1 on Windows 10. I have a few quick actions defined for keyboard shortcuts and a few sensors. Nothing interesting from what I can see in the logs. Happy to provide any additional information!

364 Replies

sreknob•7mo ago

May be related to https://discord.com/channels/1173033284519862392/1221215075117826048/1221215075117826048 but no note of persistent high CPU usage there. Attempts to restart from within the agent hang.

sreknob•7mo ago

I sort of lied, I just went spelunking though the logs again and found this a couple of time:

2024-03-13 09:04:53.650 -04:00 [FTL] [AUDIO] [audio] Error while fetching audio info: Element not found. (0x80070490)
System.Runtime.InteropServices.COMException (0x80070490): Element not found. (0x80070490)
   at HASS.Agent.Shared.HomeAssistant.Sensors.GeneralSensors.MultiValue.AudioSensors.HandleAudioOutputSensors(String parentSensorSafeName, String deviceName) in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\HomeAssistant\Sensors\GeneralSensors\MultiValue\AudioSensors.cs:line 123
   at HASS.Agent.Shared.HomeAssistant.Sensors.GeneralSensors.MultiValue.AudioSensors.UpdateSensorValues() in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\HomeAssistant\Sensors\GeneralSensors\MultiValue\AudioSensors.cs:line 191
2024-03-13 09:06:35.240 -04:00 [FTL] [AUDIO] [audio] Error while fetching audio info: Element not found. (0x80070490)
System.Runtime.InteropServices.COMException (0x80070490): Element not found. (0x80070490)
   at HASS.Agent.Shared.HomeAssistant.Sensors.GeneralSensors.MultiValue.AudioSensors.HandleAudioInputSensors(String parentSensorSafeName, String deviceName) in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\HomeAssistant\Sensors\GeneralSensors\MultiValue\AudioSensors.cs:line 167
   at HASS.Agent.Shared.HomeAssistant.Sensors.GeneralSensors.MultiValue.AudioSensors.UpdateSensorValues() in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\HomeAssistant\Sensors\GeneralSensors\MultiValue\AudioSensors.cs:line 191

2024-03-13 09:04:53.650 -04:00 [FTL] [AUDIO] [audio] Error while fetching audio info: Element not found. (0x80070490)
System.Runtime.InteropServices.COMException (0x80070490): Element not found. (0x80070490)
   at HASS.Agent.Shared.HomeAssistant.Sensors.GeneralSensors.MultiValue.AudioSensors.HandleAudioOutputSensors(String parentSensorSafeName, String deviceName) in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\HomeAssistant\Sensors\GeneralSensors\MultiValue\AudioSensors.cs:line 123
   at HASS.Agent.Shared.HomeAssistant.Sensors.GeneralSensors.MultiValue.AudioSensors.UpdateSensorValues() in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\HomeAssistant\Sensors\GeneralSensors\MultiValue\AudioSensors.cs:line 191
2024-03-13 09:06:35.240 -04:00 [FTL] [AUDIO] [audio] Error while fetching audio info: Element not found. (0x80070490)
System.Runtime.InteropServices.COMException (0x80070490): Element not found. (0x80070490)
   at HASS.Agent.Shared.HomeAssistant.Sensors.GeneralSensors.MultiValue.AudioSensors.HandleAudioInputSensors(String parentSensorSafeName, String deviceName) in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\HomeAssistant\Sensors\GeneralSensors\MultiValue\AudioSensors.cs:line 167
   at HASS.Agent.Shared.HomeAssistant.Sensors.GeneralSensors.MultiValue.AudioSensors.UpdateSensorValues() in D:\a\HASS.Agent\HASS.Agent\src\HASS.Agent\HASS.Agent.Shared\HomeAssistant\Sensors\GeneralSensors\MultiValue\AudioSensors.cs:line 191

and a couple more when restarting with additional logging turned on...

sreknob•7mo ago

2024-03-24_HASS.Agen...

sreknob•7mo ago

Here are my sensors and actions configurations, nothing fancy...

sreknob•7mo ago

quickactions.json

sensors.json

DrR0x•7mo ago

@Amadeo More memory issues

Amadeo•7mo ago

this is not related to this nor any "issue" per se what is the average increase of the consumed ram? like in mb per minute/hour, rough numbers ofc

Amadeo•7mo ago

reference sensors/commands on my side:

commands.json

sensors.json

Amadeo•7mo ago

and start of the test for me not to forget the numbers 😄 https://b.chihi.ro/Xtr4F5.png

sreknob•7mo ago

I think what's different for me is that CPU usage very seldomly goes below 8%. The increase in RAM doesn't seem linear. Been running since restart yesterday when I posted and it's only crept up slightly to 230 mb. and as I hit enter on that message, RAM usage went down to 196 MB and CPU dropped 2 % or so. It still bounces around but seems to eat significantly more CPU cycles than everything else on my system.

sreknob•7mo ago

This is a representative example of my task manager - agent just keeps chugging away at the top of CPU usage.

barrelltitor•7mo ago

I think it might possibly have to do with sensors erroring, as my screenshot sensors also had a bunch of errors when this happened Turning off my 2nd monitor immediately increased RAM usage from 200MB to 500MB The RAM usage did go down after 10-15 seconds though, and most importantly I did not look over the code so I could be waaay off, just trying to help with any info I can see on my end

barrelltitor•7mo ago

here's it using more than my firefox with 400+ tabs in terms of CPU and showing how the memory spikes way too much when just turning off a monitor, also when I turn it back on

sreknob•7mo ago

Forgot to mention, I do have the process dumps for both times there were GBs of RAM used, if you're interested in seeing them.

Amadeo•7mo ago

ohhhhh yas please I must say that I'm highly interested in both of those reports, doing a pure memory leak is ?hard? in c# with the garbage collector so there must be some interesting reason behind it same for the random 10%+ of cpu usage

DrR0x•7mo ago

That was my thinking 😂 Especially with the 20gb one!

barrelltitor•7mo ago

its currently using 16GB RAM, anything I can do to help debug? dumping the process, its taking a bit of time though The windows button doesn't open the start menu anymore when this happens lol got to 31GB ram, the dump might be quite large

barrelltitor•7mo ago

uuh is this 120GB

barrelltitor•7mo ago

here's the threads too, my system is starting to slow odown really bad i might not even be able to make the dump before it freezes or something, not sure how the memory is rising even though the process is suspended

barrelltitor•7mo ago

got to 40GB usage, the dump file is around 100GB right now

barrelltitor•7mo ago

procexp64 and task manager froze, but got some more info

barrelltitor•7mo ago

having weird graphical issues

barrelltitor•7mo ago

here's a video of some running threads Sorry for the spam sreknob, just caught this live and trying to gather as much data as i can before needing to kill it

barrelltitor•7mo ago

here's a video of the "Handles" tab in process explorer, i assume those are open file handles and other things being accessed by hass agent, looks like there's an incredible amount of audio things being used, as well as a lot of threads from hass agent https://x.titor.me/yEzE3/HOJoLaLo03.mp4

procexp64_GTgTKZdJDg.mp4 (74.51 MB)

Date: 2024-03-27 02:41:52

barrelltitor•7mo ago

@Amadeo @DrR0x sorry for the tag, if either of you are around and want me to do something specific let me know, as it's happening live I'll try to keep it running for as much as I can just in case

sreknob•7mo ago

no spam at all! we've got some interesting thing happening! here are my process dumps, if they're helpful!

sreknob•7mo ago

https://www.dropbox.com/scl/fi/v5gx5457guc7xnorflza6/HASS.Agent.DMP.gz?rlkey=278a0k10qwnn2v2j423entc31&dl=0

Dropbox

HASS.Agent.DMP.gz

Shared with Dropbox

sreknob•7mo ago

https://www.dropbox.com/scl/fi/lkkop9mhhocw9r0i6ah38/HASS.Agent.DMP-2.gz?rlkey=nhlqp6gkxpd9p2obotl9ig4wr&dl=0

Dropbox

HASS.Agent.DMP-2.gz

Shared with Dropbox

sreknob•7mo ago

FWIW, my memory usage has gone from about 300MB to 850MB from this morning. CPU still chugging away. And up to 1.7GB this morning.

sreknob•7mo ago

A lot of cpu cycles being taken by "ntdll.dll" to release cleanup.

sreknob•7mo ago

0x0000000000000000

AUDIOSES.DLL!DllGetClassObject+0x6b27
AUDIOSES.DLL+0x7835
AUDIOSES.DLL+0x7752
MMDevApi.dll!Ordinal10+0x544
ntdll.dll!RtlDeactivateActivationContext+0x2c9
ntdll.dll!TpReleaseCleanupGroupMembers+0xada
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

AUDIOSES.DLL!DllGetClassObject+0x6b27
AUDIOSES.DLL+0x7835
AUDIOSES.DLL+0x7752
MMDevApi.dll!Ordinal10+0x544
ntdll.dll!RtlDeactivateActivationContext+0x2c9
ntdll.dll!TpReleaseCleanupGroupMembers+0xada
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21

Amadeo•7mo ago

haven't read that all but I already see a lot of info that can be usefull thank you all ❤️ I'll finish my work and get to analyzuing this in the evening 🙂 after a quick glance the main suspect is/are the audio sensors and its code those dumps were taken during "normal" HASS.Agent operation?

Amadeo•7mo ago

https://b.chihi.ro/q9ul1a.png

Amadeo•7mo ago

if you say so VS

Amadeo•7mo ago

https://b.chihi.ro/Cn5Efo.png

Amadeo•7mo ago

https://b.chihi.ro/eOco3Z.png

Amadeo•7mo ago

there are more but not counted in XXk

barrelltitor•7mo ago

Should I upload my 120gb dump? It filled up my page files and my ram entirely lol, was a sight to behold I think it's too big for proper analysis or? Can it even load lol

Amadeo•7mo ago

my pc is going to regret it but yes please 😄

barrelltitor•7mo ago

Ooh sure I'm on my way back from the dentist

Amadeo•7mo ago

I have 32gb of ram on this specific machine either it loads or I need an upgrade 😄 analysing this is more difficult and interesting than I thought what is funny is that the 4gb dump says that only +-300mb are for the managed object possible that the leak is in some of the low level libs we are using? we need to figure out a way for me to replicate this somehow then I can plug in the profiler and see right away what is going on hm, this also hints towards something one thing, there is an arrow next to the HASS.Agent (Child processes) could you please check next time what hides there?

barrelltitor•7mo ago

i did, just the GUI All I had open were my sensor & command config menus, which I thought might trigger it, not sure if related tho The last time it happened I also had those menus open. It has not yet happened without the menus open. It might be 100% unrelated, but figured to mention everything as this is a strange bug

Amadeo•7mo ago

ok, that's good I mean bad but if the child ones were the UI ones then good

Amadeo•7mo ago

HASS.Agent is trying to piss me off https://b.chihi.ro/WeJiyH.png

Amadeo•7mo ago

I have a request for both of you @sreknob @barrelltitor (inconvenient but will allow us to quickly narrow down the search): 1. screenshot/backup current sensor/command config 2. remove half of the sensors/commands 3. repeat until issue is no more basically bisecting the config I have a sneaky suspicion the reason is the audio sensor but can't test any of the "fixes" until I can either reproduce the leak or I have confirmation and then I do blanket review of audio sensor doing as many changes as possible and then giving you a test version

barrelltitor•7mo ago

Yeah I'd think so too, scrolled for 10 seconds without a scroll resistance thru audio registries in the video above, it was the only thing That spammed Will check but hard to reproduce really Also op and I only share the audio sensor I believe Yeah I don't have any sensors he has other than audio

Amadeo•7mo ago

the dumps (despite stating that managed "dead" object are around 300mb) also at least suggest that so I'd propose removing only audio sensors and audio commands first and then seing if it will run "suspiciously long" without any issue if this will be the case, fix will be quite simle, there are few places where CoreAudio library is not interacted properly I think because for now I assume mine/Sam's fault and not library's

Amadeo•7mo ago

https://b.chihi.ro/PunFff.png

Amadeo•7mo ago

In the meantime I'll try to create some artificial environment to execute audio sensor and commands code to see how it behaves

barrelltitor•7mo ago

I've been speeding up and slowing down a plex video with a browser extension, will try again forgot to upload the dump, still need it or?

Amadeo•7mo ago

if you have disk space please keep hold to it in case but looking at the fact that both dumps pointed kinda towards audio being the issue I'd say let's focus on confirming that + no 100% certainty that my poor system will be able to load it 😄 I need to get 128gb of ram just to brag about it

barrelltitor•7mo ago

idk how it even got to that point, I don't have 128GB ram lol well, page size can easily get there I guess i've tried taking all the steps I have again and it hasn't happened again after removing just the audio sensor, though I want to test for longer to be sure

sreknob•7mo ago

So far, so good with the audio sensor turned off (rest of my config the same). Needed to kill the process after storing the sensor config with the audio sensor disabled but immediately CPU back down to 0-0.1% and RAM stable (108 MB). Will keep you posted with changes!

Amadeo•7mo ago

thank you both for the update, please keep an eye on it and I'll start reworking/auditing all audio sensors/commands interesting I have gutted the sensor and created an environment where it's constantly updated two observervations: - it's resource intensive, one udate takes +-0.5second - ram usage is steadily growing and I already see a place where general audiosensor performance can be increased the c# profiler is nice thingy and it's starting to freze sometimes 😄 and now stuck at 280mb for some time, more interesting with every second

Amadeo•7mo ago

now I'm going to do some house chores, wonder how much there will be once back 😄 https://b.chihi.ro/P16Xqz.png

Amadeo•7mo ago

https://b.chihi.ro/vNiV1w.png

Amadeo•7mo ago

so yeah so, I've created a dirty simplest of cases:

            var AudioDeviceEnumerator = new MMDeviceEnumerator(Guid.NewGuid());

            while (true)
            {
                foreach (var audioDevice in AudioDeviceEnumerator.EnumerateAudioEndPoints(DataFlow.Render, DeviceState.Active))
                {
                    var name = audioDevice.DeviceFriendlyName;
                    audioDevice.Dispose();
                }
            }

            var AudioDeviceEnumerator = new MMDeviceEnumerator(Guid.NewGuid());

            while (true)
            {
                foreach (var audioDevice in AudioDeviceEnumerator.EnumerateAudioEndPoints(DataFlow.Render, DeviceState.Active))
                {
                    var name = audioDevice.DeviceFriendlyName;
                    audioDevice.Dispose();
                }
            }

and the ram usage still seems to be going up not particularly good

Amadeo•7mo ago

I'll crosscheck this with another (alternative?) audio lib https://github.com/naudio/NAudio

GitHub

GitHub - naudio/NAudio: Audio and MIDI library for .NET

Audio and MIDI library for .NET. Contribute to naudio/NAudio development by creating an account on GitHub.

Amadeo•7mo ago

https://github.com/ho0ber/NK2Tray/issues/60

GitHub

Memory leak containing NAudio.CoreAudioApi items · Issue #60 · ho...

Describe the bug There's a memory leak attributed to the use of naudio/NAudio somewhere. I haven't been able to find where, but every time the pop-up menu is opened, the same amount of &quo...

Amadeo•7mo ago

Nice

Amadeo•7mo ago

yup, https://b.chihi.ro/0haAFx.png

Amadeo•7mo ago

same

barrelltitor•7mo ago

oh damn nice find Oh one more thing that might be related, with the audio sensor on sometimes I had crackling audio I have an audio interface & studio monitors, usually the audio crackling for these was due to cpu clock dropping due to power settings or viewing my screens with rustdesk I attributed this to the RAM being filled up & causing all sorts of oom issues though It wasn't constant though, just in random times, and since you said it took a long time to get audio data maybe it somehow was causing it After removing the audio sensor I haven't seen it use 10% cpu usage anymore, at most 0.2%

Amadeo•7mo ago

https://b.chihi.ro/LzWuDQ.png

Amadeo•7mo ago

the NAudio is much more stable than the current one (CoreAudio) 319 minutes I've noticed the cpu load increase as the memory increased also with CoreAudio

Amadeo•7mo ago

https://b.chihi.ro/DvVbWp.png

Amadeo•7mo ago

running since morning (17:52 for me now) NAudio leaks but slower I'll do some more tests and then either go with it (and I'll try to find a way to release that memory) or open a bottle of nice honey mead and go the p/invoke route going to be fun looks like "AudioSwitcher" doesn't leak memory as easily (at least from the small test I did) downside is that it was really last updated years ago (on the other side wasapi also probably didn't change since it's windows sooo) I'll give it a broader go an play with it and it doesn't give access to AudioEndpointVolume

Amadeo•7mo ago

https://tenor.com/view/racoon-screaming-aaaaaaaaaaaaa-rage-gif-27241414

Tenor

Amadeo•7mo ago

https://github.com/xenolightning/AudioSwitcher/issues/42

GitHub

Can I use GetMasterVolumeLevelScalar with this API? · Issue #42 · x...

I have downloaded AudioSwitcher.AudioApi.CoreAudio using NuGet. I was then trying to use GetMasterVolumeLevelScalar to get the Windows volume. However, I don't find this function in the Api. Co...

Amadeo•7mo ago

god give me strength oh, maybe I can work around it sorry for spamming, helps me think 😄 ok, this looks promising, there is no support for peak volume yet but plugging in AudioSwitcher instead of CoreAudio looks OK and updates hell of faster

Amadeo•7mo ago

ffs https://github.com/xenolightning/AudioSwitcher/issues/34

GitHub

Known Issue: Audio Sessions - "Random" exceptions · Issue #34 · xen...

There is an issue in the session create code which causes either COM Objects to be access in a non thread safe manner, or the lifecycle is disposed prematurely. The session controller should probab...

DrR0x•7mo ago

Damn, that's a rip

Amadeo•7mo ago

worked around it KINDA @barrelltitor @sreknob would you be ok downloading a test build from github (my dev fork)? I'll replace CoreAudio with AudioSwitcher lib because atm that's the best we can do, I can see that sometimes mem usage spikes up when I launch a new program or something (the artificial code to run sensor code as fast as possible) using audio but in general it doesn't go up by itself

Amadeo•7mo ago

https://b.chihi.ro/8v8xxB.png

Amadeo•7mo ago

+ it's on 615k iterations so

barrelltitor•7mo ago

yeah

Amadeo•7mo ago

I'll get to work, should be ready today/tomorrow I hope 😄

barrelltitor•7mo ago

does the verbose logging show absolutely everythign it does? if not, can we get an even verboser loogging? im thinking if we can isolate what's going on, can just monitor RAM with the HA sensor and check the logs when it starts going haywire(if it happens again)

Amadeo•7mo ago

regarding the leak of original (CoreAudio) lib unfortunately no afaik each time certain functions are called there is an orphaned com object created that cannot be freed by GC if my investigation was correct ok, it was faster than I thought, still need to implement the peak volume somehow as this lib handles it differently but looks to be working

barrelltitor•7mo ago

it gets cleared up sometimes hmm either that or gets pushed in the pagefile and i just dont notice At one point it went from 40GB RAM usage to 20GB

Amadeo•7mo ago

40gb 😮

Amadeo•7mo ago

move to AudioSwitcher has been easy enough with one downside, due to a lib bug there will be no support for "session peak volume" but only "peak volume" in general https://github.com/xenolightning/AudioSwitcher/pull/63

GitHub

avoid unnecessarily triggering STA COM thread check by pengowray · ...

This is another tiny patch. it's also a little lazy: I haven't got the library to compile yet so haven't checked if it 100% fixes the problem. This time it's for the most recent rel...

Amadeo•7mo ago

but I'll do some additional research before pulling the trigger

sreknob•7mo ago

Awesome, thank you and I'll be happy to test! Can you link me to the branch that has the change? I didn't see it looking at recent merges.

Amadeo•7mo ago

the branch is https://github.com/amadeo-alex/HASS.Agent/tree/fix-audio-memleak

GitHub

GitHub - amadeo-alex/HASS.Agent at fix-audio-memleak

DEV PLATFORM - go to https://github.com/hass-agent/HASS.Agent for downloads :) - GitHub - amadeo-alex/HASS.Agent at fix-audio-memleak

Amadeo•7mo ago

not merged anywhere yet as I'm not done fully testing I'll push some more changes that I did and then try to build it for you via gh actions

Amadeo•7mo ago

I have HASS.Agent running for some time and it so far it has gained 4mb per VS monitoring and is at 204mb - the task manager says something else though 😄 https://b.chihi.ro/Mj7Zgf.png

Amadeo•7mo ago

@sreknob @barrelltitor <REMOVED> please backup your current installation or at least configuration files not that I expect anything breaking but it's just a healthy habit 😄 argh I failed, please give me a sec 😄 for some reason it didn't build the test branch, lemme fix it

Amadeo•7mo ago

https://github.com/amadeo-alex/HASS.Agent/releases/tag/2.0.1-alpha-audioleakfix

GitHub

Release 2.0.1-alpha-audioleakfix · amadeo-alex/HASS.Agent

Please do not download files from this "release" It's only purpose if to provide testing bed for the insiders/issue reporters. Unless explicitly instructed by the developers, please d...

Amadeo•7mo ago

now it's correct 😄

barrelltitor•7mo ago

nice, will try it out tonight

barrelltitor•7mo ago

nice

Amadeo•7mo ago

Not directly related but I've received answer to one of the bug issues created regarding this, interesting one to say the least, I'll play with this approach (I'd like to keep the library if possible) https://github.com/morphx666/CoreAudio/issues/27#issuecomment-2030653792

GitHub

Bug: possible memory leak · Issue #27 · morphx666/CoreAudio

Hello, I've been experimenting with various "CoreAudio" wrapper libraries and it looks like most of them, including this one, appear to be leaking memory - slowly but constantly (or i...

barrelltitor•7mo ago

oh that sounds like a good way to fix it if you want i can wait for you to try that way and I can test it i can test the build above too, but figured if you want to keep the library might wait to see if it can be fixed first

Amadeo•7mo ago

even with this proposal the leak appears to be still there (☞ﾟヮﾟ)☞ so I'd recommend testing the changed build 😄 @cannonbell jenkins you can try this version although remember this please ^ 😄

cannonbell jenkins•7mo ago

i shall as soon as i find out how to create a backup hahah if im not mistaken ois this about a memory leak or something? could this be attributed to my other applications also crashing? (discord/opera gx)

Amadeo•7mo ago

if by some reason hass.agent would eat up all your memory then possibly the most important are "config" folders of both client and satellite service

cannonbell jenkins•7mo ago

sorryt where do i find these?

barrelltitor•7mo ago

really sorry, have been caught up with work stuff and im exhausted, will test first thing tomorrow

Amadeo•7mo ago

default ones would be: service - C:\Program Files\HASS.Agent\Service\config client - C:\Users<yourusername>\AppData\Local\HASS.Agent\Client\config no worries 🙂

barrelltitor•7mo ago

I think it's going to be next to your exe, you can search in the windows bar then right click -> open file location then do the same for the shortcut you find there

Amadeo•7mo ago

some new revelations regarding issue on CoreAudio (giving hope :D) either way if you all can please test the one that I've posted before just in case

barrelltitor•7mo ago

i have to test idk if related at all or if im just imagining it but changing my volume seems a bit laggy now ah no it solved itself after i played around with the audio volume a bit I have an ikea rotary encoder volume thing that's really nice to use, changes volume by 2 each step and it's smooth, was lagging very badly when I initially restarted cpu and ram have been quite normal with the new version

Amadeo•7mo ago

Don't give me hopes 🥲 I'll also try to whip up a CoreAudio "nonleaky" version if you'd be up to testing that also

barrelltitor•7mo ago

<200mb tops so far iim happy to test that too one nice thing is that before the CPU used to go even up to 10%, now it stays in a steady 0%-0.3%ish oh im an idiot could you maybe put sensors for hass agent's cpu and ram usage? that way we could way more easily monitor it lol

Amadeo•7mo ago

internally HASS.Agent is using LibreHardwareMonitor so that shouldn't be hard

barrelltitor•7mo ago

worked fine so far, I'll be happy to test the coreaudio version as well when it's ready

barrelltitor•7mo ago

does the setapplicationvolume command not work on this new testing version?

barrelltitor•7mo ago

i didnt actually check to see if the audio sensor was working in the new version lol. the audio sessions sensor isn't working and set application volume isn't either

barrelltitor•7mo ago

Restarting it fixed it woo

Amadeo•7mo ago

sus that is stopped/didn't work

barrelltitor•7mo ago

i'd rather just assume it's my fault or not the fault of the software unless i can actually reproduce the issue Ah found something, before it used to show inactive audio sessions too, now it shows just the ones playing

barrelltitor•7mo ago

not sure how hass agent has an audio session though

barrelltitor•7mo ago

or are all sessions by definition active and the inactive ones are something else? Though I am sure that list showed inactive ones too, can revert to test

Amadeo•7mo ago

Could be both, I remember it showing active only but it's easy enough to change I should be able to get the CoreAudio this weekend unless BBQ life gets better of me 😄

barrelltitor•7mo ago

bbq > coreaudio

Amadeo•6mo ago

I swear when I finish with this I'm not touching aduio stuff for a year 😄 now CoreAudio is not liking the fact that the audio devices are gathered initially from UI thread and accessed later from background thread I don't want to jinx it but I think I finally found a library with almost the same functionality as CoreAudio which may not leak memory just because you're looking at it further tests to come or not, ffs, I hate audio stuff 😄 attampt 2137, got CSCore test setup to work, it goes up in ram usage but so far it's being GC-ed and stays between 60mb and 90mb

Amadeo•6mo ago

https://b.chihi.ro/6z8uoH.png

Amadeo•6mo ago

🤞 you might think that you know what beauty is you don't

Amadeo•6mo ago

https://b.chihi.ro/SGZSJn.png

Amadeo•6mo ago

this yellow marker is (☞ﾟヮﾟ)☞ I feel like each time I make a step forward umanaged code is like

nah m8, won't be that easy

I swear I learned more about memory mgmt due to those bug than in my entire life

Amadeo•6mo ago

https://b.chihi.ro/xkAfNb.png

Amadeo•6mo ago

if I'm reading this right getting friendly name from the audio device causes heap memory leak the f 😄

barrelltitor•6mo ago

I have not touched C since high school, I only work with languages that don't let me touch memory so this might sound very silly, but would it be possible to restrict the amount of memory for audio gathering and just wipe it all regularly when a check isn't happening? sounds like it's more the library/windows having issues than how you're using it im happy to test anything ofc

Amadeo•6mo ago

not in a simple manner, if .net GC is not able to collect something it's either because some other thing still keeps a reference to it or you in one way or another done "gimme X memory m8" which .net allocates and gives you but then you forget to "give it back" so far the least leaky one is CSCore although I have NAudio on my list for today I have 7 days of trial on this marvelous memory profiler 😄 gotta make the best of it + I'll see if CSCore is easily fixable (if it doesn't exceed my low level skills)

Amadeo•6mo ago

https://b.chihi.ro/bK0APN.png

Amadeo•6mo ago

🥲 what in the c#'s love of god

Amadeo•6mo ago

https://b.chihi.ro/EKoMyw.png

Amadeo•6mo ago

https://b.chihi.ro/lMpToD.png

Amadeo•6mo ago

spot an issue game @JonnyBergdahl you'll probably like it xD or correct me since you have more c# experience

JonnyBergdahl•6mo ago

Wait what? Never seen that before! I only ever used the IDisposable pattern.

Amadeo•6mo ago

I think that's a IDisposable/2 some classes of this lib implement it properly at least this one doesn't 😄

almost disposable

JonnyBergdahl•6mo ago

"Yak". 🙂

Amadeo•6mo ago

tell me you have a memleak without telling me you have a memleak https://b.chihi.ro/uIwYZP.png

Amadeo•6mo ago

nope, NAudio is not worth the time

JonnyBergdahl•6mo ago

So beautiful! Just add more RAM. It works for Google Chrome!

Amadeo•6mo ago

https://downloadmoreram.com/

DownloadMoreRAM.com - CloudRAM 2.0

Amadeo•6mo ago

I'll need it to run with the profiler at least 1h+ but promising (idk which time again I'm saying this xD)

Amadeo•6mo ago

https://b.chihi.ro/3FLtTf.png

Amadeo•6mo ago

for anyone that might looks at those messages way later: CSCore with "FriendlyName" caching

barrelltitor•6mo ago

what does the graph represent tho like is that 200mb or does it go to gigabytes lol

Amadeo•6mo ago

this is managed (orange) and unmanaged (red line) memory visualization the "sawtooth" line is what we want to see, resources are allocated when needed and once every X GC comes in and cleans stuff up the important thing is that the unmanaged memory usage (red line) doesn't seem to be just so slightly increasing with every "iteration" at least so far 😄

Amadeo•6mo ago

https://b.chihi.ro/xO0HTA.png

Amadeo•6mo ago

maybe I'm going a bit to far with the optimizations of audio mgmt now but damnn that stright red line is nice 😄 now I need some long running task to see if I haven't forgotten about anything ok, got the sensor part fully ready - on events which speeds the code quite a bit now to get the commands working and move the code to the test build now I'm not certain if CSCore has the ability to set default endpoints I think I need a walk a touch of grass so to say... ok, I jumped through that many hoops, I'll jump that last one @barrelltitor do you still have win10?

barrelltitor•6mo ago

yeah

Amadeo•6mo ago

ok, device chaning code is ready - DIY'd it and it doesn't look like it's leaking memory (☞ﾟヮﾟ)☞ gonna put it in the hass agent and do a test built - should be ready tomorrow (it's 23:00 here) ok, I have the build ready but please give me one more day - I want to iron out as many possible bugs as possible before handing it to you sorry for the false promise for today but I don't want you to be testers for common bugs but rather for more complicated scenarios 😄

Amadeo•6mo ago

on the positive note @barrelltitor I have a screenshor for you regarding https://github.com/hass-agent/HASS.Agent/issues/72

GitHub

Feature: Set application volume for different audio streams of the ...

Is your feature request related to a problem? Please describe. I am trying to replicate deej in home assistant as I already have a bunch of wireless audio controlling devices I can use, as well as ...

Amadeo•6mo ago

https://b.chihi.ro/yhz7Rp.png

Amadeo•6mo ago

I didn't change them manually just to be clear 😄

barrelltitor•6mo ago

That is amazing ❤️ thank you so much i'm in no hurry

Amadeo•6mo ago

@barrelltitor (and @sreknob @cannonbell jenkins if you'd also like to try 🙂 ) https://github.com/amadeo-alex/HASS.Agent/releases/tag/2.0.1-alpha-audioleakfix-alt

GitHub

Release 2.0.1-alpha-audioleakfix-alt · amadeo-alex/HASS.Agent

Please do not download files from this "release" It's only purpose if to provide testing bed for the insiders/issue reporters. Unless explicitly instructed by the developers, please d...

Amadeo•6mo ago

I tried to test all audio stuff and didn't notice any stright in my face issues - let me know if you encounter any

barrelltitor•6mo ago

I'm sorry, I have the worst luck in human history. Both of my monitors died. Lucky rustdesk auto starts with my computer. Tomorrow is labor day, so I will have to wait until after that to see if I can fix or replace them My keyboard works, and with rustdesk my phone is like a tiny monitor, but it's very difficult to do anything lol

Amadeo•6mo ago

what the f both at the same time?

DrR0x•6mo ago

Yeah what? How Worst luck ever or something?

rafo•6mo ago

I've been running the new alpha for a bit, and just received a crash. I'm not sure it's related. Here's the logs: https://pastebin.com/angVa1tj It happened after HA TTS proxy was sending some audio to play (on the PC).

Pastebin

024-05-03 17:11:22.227 +02:00 [INF] [MEDIA] Received media: http://...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

rafo•6mo ago

https://pastebin.com/BiMLi8ET

Pastebin

error #1:Application: HASS.Agent.exeCoreCLR Version: 6.0.2924.17105...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

rafo•6mo ago

random (silent) crashes used to also happen also with the non-alpha version - I always suspected the audio sensors

Amadeo•6mo ago

kernelbase.dll

Ffs Thank you for the logs, I'll play around with tts to see if I can reproduce this Was it a one time thing or it's constsnt with each of tts calls?

rafo•6mo ago

I can't replicate it, seems like a one-time thing. I'll keep an eye out

Amadeo•6mo ago

How long was the tts message? Rougly ofc 😄 I'm trying to picture out scenario where this situation could occurr @rafo I'm not sure in what time zone you are but I'll whip up a custom build with additional checks around disposal of audio objects today (23:15 here atm) or tomorrow to see if we can handle those edge cases somehow

Amadeo•6mo ago

@rafo https://github.com/amadeo-alex/HASS.Agent/releases/tag/2.1.0-beta2-audiodispose (note: this test build is based on 2.1.0-beta2 so it includes other changes/features)

GitHub

Release 2.1.0-beta2-audiodispose · amadeo-alex/HASS.Agent

Please do not download files from this "release" It's only purpose if to provide testing bed for the insiders/issue reporters. Unless explicitly instructed by the developers, please d...

rafo•6mo ago

Thanks, I'll install it today, and let you know if I see any crashes. Is there anything I should look out for?

Amadeo•6mo ago

let it run for some time (with extended logging enabled) and then after a day or two look into the logs for "[AUDIOMGR] Exception disposing" if it's going to be there and there wasn't any functionality break/crash then good, it's doing its job if not and there will be additional crash - I have a problem 😄

barrelltitor•6mo ago

I don't know, been having lots of power issues in the block forever, sometimes the electricity company does the "turn it off and on" thing multiple times a second and things get weird I have a UPS specifically for this, so I don't understand what's going on. Have to wait a couple more days to get them to a repair shop due to holidays, but I did get a monitor to use from a buddy for now the installer killed the satellite agent process properly, but it thought it didn't, might need an extra check or delay or something

Amadeo•6mo ago

as we speak I'm redesigning the installer (separating service from client for more installation options - non admin one for example), should also fix this one I'd flood them with complaints and repair bills of the destroyed equipment

barrelltitor•6mo ago

one thing I noticed without any new sensors or commands is that ram usage is higher than before, about double. Used to be around 100MB now it's around 170-200MB Not sure if that's an issue. It's not growing, I actually just saw it go from 200MB slowly to 160MB after I closed the GUI These are still very small numbers though, doesn't look like it's leaking for now opening up the GUI is about 20MB of RAM so now it went to 180mb, it's very sowly climbing Now it's at 190MB I don't see it going over 230~MB RAM in peaks. Been watching it for a few minutes. RAM is higher than before(160MB+ compared to 100MB), and it does seem to dance more around 160MB~230MB, but it doesn't spiral out of control, it chills out and goes back down to 160MB Will keep an eye out to see if it ever goes above, but it seems good so far

Amadeo•6mo ago

might be a bit higher due to the fact that "audio stuff" is cached now - we do as little of audio operations as possible (runtime benefit at expense of ram + less posibilities for memleak) + there are additional changes with the 2.1.0 builds (like NFC support) that may use X or Y of ram due to m$ libraries loaded but I'll take a look if we can cut that down somehow

barrelltitor•6mo ago

I don't think 60mb is an issue at all, it might actually be less of a difference as it probably used more than 100MB that was just the average, just figured to let you know of everything I see If it's normal and expected then we're golden nothing to update for now, it's still around 200MB~ish, hasn't really gone up

rafo•6mo ago

@Amadeo here's a fresh (crash) log from today: https://pastebin.com/q3qFWF5E This happened after about half an hour after I woke the computer up from sleep.

Pastebin

2024-05-06 10:49:35.023 +02:00 [INF] [SYSTEMSTATE] Session resuming...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

Amadeo•6mo ago

Damnnit, I hope this is not occuring internaly within the library Were there any other tts messages in that 30 minutes between waking up the computer an the crash or was it the first one and it crashed on it?

rafo•6mo ago

I think this was the first one, but I'm not 100% sure. It's an automated reminder from HA - that tells me not to be too close to the monitor (based on a TOF sensor). ah, based on HA logs - this was the first notification of the day

Amadeo•6mo ago

lol, I want that so we have a potential fault point sleep->wake->tts do you have sleep or hibernation configured? asking as I want to try to reproduce that on my side

rafo•6mo ago

just a plain s3 sleep also a note, this was the 2nd wake-up from sleep since I started hass.agent, and I'm on windows 10 (latest patches)

barrelltitor•6mo ago

i have 0 cluee what the ram usage is based on but today the agent is using 70MB ram, lower than ever in the history of me using it I'd say there's no more leak

barrelltitor•6mo ago

Amadeo•6mo ago

don't jinx it 😄

barrelltitor•6mo ago

one big issue I didn't notice, it's not connecting to the HA integration anymore

barrelltitor•6mo ago

Amadeo•6mo ago

is that the media control?

Amadeo•6mo ago

like this one:https://b.chihi.ro/Q59c8h.png

Amadeo•6mo ago

and are you running HA 2024.5.X?

Amadeo•6mo ago

because if so: https://github.com/hass-agent/HASS.Agent/issues/75#issuecomment-2090094046

GitHub

Bug: Latest HA Core update 2024.5.0 breaks notification entities. ·...

Please make sure that the issue you're planning to report is not already registered :) Describe the bug The latest HA Core update (2024.5.0) stops hass.agent from creating any notify entities. ...

barrelltitor•6mo ago

i see you need to code tag the should be `init__.py`

Amadeo•6mo ago

Ah, ya, good spot I'll fox it tmrw, thank you

barrelltitor•6mo ago

It looks like instead of a memory leak, now the memory gets lower the more I use it, it's at 60MB right now I don't know what sorcery you've done, but this is incredible optimization

Amadeo•6mo ago

is there anything new with this? I'l still trying to somehow replicate it

rafo•6mo ago

It keeps happening 😅, I figured the logs are the same, so no point spamming..

Amadeo•6mo ago

Ok, understood

rafo•6mo ago

I have VS2022 installed, if there's a way to collect more debug info for you, I can run something in the background and wait until it crashses

Amadeo•6mo ago

Hmm, worst case scenario I'll tell you how to run it via vs and then it should catch the crash 😄 Good to know, how accustomed with vs are you?

rafo•6mo ago

I used to code C# for my day job - like 7 years back 😄 with some instructions, I should be able to collect some data

Amadeo•6mo ago

the easiest thing to try would be to attach the VS debbuger to running instance of HASS.Agent

Amadeo•6mo ago

then as a test you can press break all to see if it'll use the debug symbols https://b.chihi.ro/QNcsH2.png

rafo•6mo ago

this look ok?

Amadeo•6mo ago

ya better than I expected tbh so run HASS.Agent normally and attach it then wait for the crash (without pressing break all), hopefully it catches it

rafo•6mo ago

    [Guid("641DD20B-4D41-49CC-ABA3-174B9477BB08")]
    public sealed class AudioSessionNotification : IAudioSessionNotification
    {
        public event EventHandler<SessionCreatedEventArgs> SessionCreated;

        int IAudioSessionNotification.OnSessionCreated([In] IntPtr newSession)
        {
            if (this.SessionCreated != null)
            {
                AudioSessionControl audioSessionControl = new AudioSessionControl(newSession);
                ((IUnknown)audioSessionControl).AddRef();
                this.SessionCreated(this, new SessionCreatedEventArgs(audioSessionControl)); --> it fails here with System.NullReferenceException: 'Object reference not set to an instance of an object.' - this.SessionCreated is null according to VS
            }
            return 0;
        }
    }
}

    [Guid("641DD20B-4D41-49CC-ABA3-174B9477BB08")]
    public sealed class AudioSessionNotification : IAudioSessionNotification
    {
        public event EventHandler<SessionCreatedEventArgs> SessionCreated;

        int IAudioSessionNotification.OnSessionCreated([In] IntPtr newSession)
        {
            if (this.SessionCreated != null)
            {
                AudioSessionControl audioSessionControl = new AudioSessionControl(newSession);
                ((IUnknown)audioSessionControl).AddRef();
                this.SessionCreated(this, new SessionCreatedEventArgs(audioSessionControl)); --> it fails here with System.NullReferenceException: 'Object reference not set to an instance of an object.' - this.SessionCreated is null according to VS
            }
            return 0;
        }
    }
}

fileName AudioSessionNotification.cs this happens right after the computer wakes up from sleep (after it was in s3 sleep overnight) here's some deets on the exception:

System.NullReferenceException
  HResult=0x80004003
  Message=Object reference not set to an instance of an object.
  Source=CSCore
  StackTrace:
   at CSCore.CoreAudioAPI.AudioSessionNotification.CSCore.CoreAudioAPI.IAudioSessionNotification.OnSessionCreated(IntPtr newSession) in CSCore.CoreAudioAPI\AudioSessionNotification.cs:line 18

System.NullReferenceException
  HResult=0x80004003
  Message=Object reference not set to an instance of an object.
  Source=CSCore
  StackTrace:
   at CSCore.CoreAudioAPI.AudioSessionNotification.CSCore.CoreAudioAPI.IAudioSessionNotification.OnSessionCreated(IntPtr newSession) in CSCore.CoreAudioAPI\AudioSessionNotification.cs:line 18

and a screenshot: the same exception was thrown again (I pressed continue in VS on the first one)

Jatoxo•6mo ago

There was a leak when using the audio sensor? I'm still on 2022.14 and I've been running for days with the audio sensor and it's definitely not leaking. Did something change or is it edge case?

rafo•6mo ago

@Jatoxo it's an edge case, that we're chasing, happens once per 2-3 days, after PC wakes up from sleep I've had problems with this even on the old 2022.14 version..

Amadeo•6mo ago

@rafo ❤️ that's some good debug information right here aaaand that's an issue within the library itself the audio stuff is going to be the main topic of my nightmares for a long time I think we have 3 options here: 1. reload audio management after waking up from sleep 2. try to force the library to use my implementation of this class (with the fix) 3. fork the library, fix it, publish a nuget and use it instead of the original

DrR0x•6mo ago

2 seems the best, thats from a python view tho 🤣 can you not just override the specific class thats going wrong? until they fix it

Amadeo•6mo ago

with reflection almost everything is possible but it's also not something that is "good" that's not how the stuff should be done 😄 sooo, I have a POC ready of audio manager being cleaned up with sleep and reinitialized with resume not pretty since the current HASS.Agent arch doesn't really allow it but looks to be working? I'll do some cleanup and then create a test build

rafo•6mo ago

whoa, tha was fast

DrR0x•6mo ago

idk how you just drop whatever you are doing and immediately full focus on Hass.Agent but props to you 🤣

Amadeo•6mo ago

boring call at work xD lol and now I've got a null reference, in completely different place, that should have been caught by try/catch god give me patience 😄

Amadeo•6mo ago

@rafo https://github.com/amadeo-alex/HASS.Agent/releases/tag/2.0.1-beta2-audio-sleep I think I've got them all

GitHub

Release 2.0.1-beta2-audio-sleep · amadeo-alex/HASS.Agent

Please do not download files from this "release" It's only purpose if to provide testing bed for the insiders/issue reporters. Unless explicitly instructed by the developers, please d...

rafo•6mo ago

Cheers! Installed it, and now I'm up and running, debugger attached. Let's see what happens. weel, it crashed again. This time it happened when * music was playing through speakers on the monitor * monitor went to sleep (this disconnected the speakers and switched output device to headset) * monitor woke up from sleep (quite quickly, as I was sitting at the pc) * speakers reconnected and music started playing from speakers again * crash

CSCore.CoreAudioAPI.CoreAudioAPIException
  HResult=0x88890004
  Message=IAudioSessionControl::RegisterAudioSessionNotification caused an error: 0x88890004, "Unknown HRESULT".
  Source=CSCore
  StackTrace:
   at CSCore.CoreAudioAPI.AudioSessionControl.RegisterAudioSessionNotification(IAudioSessionEvents notifications) in CSCore.CoreAudioAPI\AudioSessionControl.cs:line 270
   at CSCore.CoreAudioAPI.AudioSessionNotification.CSCore.CoreAudioAPI.IAudioSessionNotification.OnSessionCreated(IntPtr newSession) in CSCore.CoreAudioAPI\AudioSessionNotification.cs:line 16

CSCore.CoreAudioAPI.CoreAudioAPIException
  HResult=0x88890004
  Message=IAudioSessionControl::RegisterAudioSessionNotification caused an error: 0x88890004, "Unknown HRESULT".
  Source=CSCore
  StackTrace:
   at CSCore.CoreAudioAPI.AudioSessionControl.RegisterAudioSessionNotification(IAudioSessionEvents notifications) in CSCore.CoreAudioAPI\AudioSessionControl.cs:line 270
   at CSCore.CoreAudioAPI.AudioSessionNotification.CSCore.CoreAudioAPI.IAudioSessionNotification.OnSessionCreated(IntPtr newSession) in CSCore.CoreAudioAPI\AudioSessionNotification.cs:line 16

when I continued with the original exception, another was thrown:

System.AccessViolationException
  HResult=0x80004003
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

System.AccessViolationException
  HResult=0x80004003
  Message=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

then it crashed completely, continue in VS did not help

Amadeo•6mo ago

interesting this sounds like a separate issue did the same scenario happen before? I mean only monitor being off

rafo•6mo ago

don't recall that it did but I did not have it attached to debugger before

Amadeo•6mo ago

now I need to find a reliable way of reproducing it I curse all audio stuff 😄 ok, I got something, not exactly what you've shown but at least something I got it once when turning the monitor on and off while music was playing through it and then quite reproducable by just disabling the monitor speakers in windows audio controlls ok, fixed at least part of it I can now turn on/off different devices and the monitors without any issues

Amadeo•6mo ago

@rafo attempt number idk which - thank you for all the help debbuging this 😄 - https://github.com/amadeo-alex/HASS.Agent/releases/tag/2.0.1-beta2-audio-sleep2 I'll do my best trying to catch all possible bugs - as long as we don't encounter a non-fixable library bug we can try to stick with CSCore

GitHub

Release 2.0.1-beta2-audio-sleep2 · amadeo-alex/HASS.Agent

Please do not download files from this "release" It's only purpose if to provide testing bed for the insiders/issue reporters. Unless explicitly instructed by the developers, please d...

rafo•6mo ago

Man, you're a legend. I'll give it a try tomorrow. We're getting there! Soo, I've got another unhandled exception:

System.Runtime.InteropServices.SafeArrayRankMismatchException
  HResult=0x80131538
  Message=SafeArray of rank 0 has been passed to a method expecting an array of rank 1.
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

System.Runtime.InteropServices.SafeArrayRankMismatchException
  HResult=0x80131538
  Message=SafeArray of rank 0 has been passed to a method expecting an array of rank 1.
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

this one has no call stack - maybe it's coming from unmanaged code? it got re-thrown multiple times, too this got logged in the output window

Exception thrown: 'System.Runtime.InteropServices.SafeArrayRankMismatchException' in CSCore.dll
An exception of type 'System.Runtime.InteropServices.SafeArrayRankMismatchException' occurred in CSCore.dll and wasn't handled before a managed/native boundary
SafeArray of rank 0 has been passed to a method expecting an array of rank 1.

Exception thrown: 'System.Runtime.InteropServices.SafeArrayRankMismatchException' in CSCore.dll
An exception of type 'System.Runtime.InteropServices.SafeArrayRankMismatchException' occurred in CSCore.dll and wasn't handled before a managed/native boundary
SafeArray of rank 0 has been passed to a method expecting an array of rank 1.

it seems to be crashing under the same conditions - monitor wakes up from sleep, while music is playing

Amadeo•6mo ago

System.Runtime.InteropServices.SafeArrayRankMismatchException

that's a new one

rafo•6mo ago

I've seen it before, but thought it was somehow connected to those exceptions you already fixed

Amadeo•6mo ago

+the lack of any stack trace is concerning because it's an enigma error without it

rafo•6mo ago

I've switched to windbg, maybe it'll show some more info i'm not really an expert, but it seems that there are not many managed methods in the CSCore library, that use arrays: https://github.com/search?q=repo%3Afiloe%2Fcscore+LPArray&type=code

Amadeo•6mo ago

I have a sneaky suspicion this might me the hoop we can't jump through at least not with CSCore

rafo•6mo ago

I'll leave it running under windbg, let's see what happens..

Amadeo•6mo ago

got NAudio lib under profiler in case we need to make a switch - with the same arch that we used fo CSCore (so event bases and not constant pooling) ok, at least NAudio doesn't seem to leak anything with the current approach - if it'll fix the monitor sleep issue is another topic tho I'm going to create a build with NAudio so we can see if it suffers the same issue as CSCore if you're up to it of course

rafo•6mo ago

ofc, happy to test 🙂

Amadeo•6mo ago

@rafo so, let's try again 😄 https://github.com/amadeo-alex/HASS.Agent/releases/tag/2.0.1-alpha-audioleakfix-naudio

GitHub

Release 2.0.1-alpha-audioleakfix-naudio · amadeo-alex/HASS.Agent

Please do not download files from this "release" It's only purpose if to provide testing bed for the insiders/issue reporters. Unless explicitly instructed by the developers, please d...

rafo•5mo ago

all right, 4th time is the charm? soo, good news/bad news: * audio stuff does look OK so far * now I'm getting crashes on the virtual desktop part (I don't have virtual desktop sensors enabled)

System.Runtime.InteropServices.COMException
  HResult=0x8007139F
  Message=The group or resource is not in the correct state to perform the requested operation. (0x8007139F)
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

System.Runtime.InteropServices.COMException
  HResult=0x8007139F
  Message=The group or resource is not in the correct state to perform the requested operation. (0x8007139F)
  Source=<Cannot evaluate the exception source>
  StackTrace:
<Cannot evaluate the exception stack trace>

after hitting continue it throws this one:

System.Runtime.InteropServices.COMException
  HResult=0x8007139F
  Message=The group or resource is not in the correct state to perform the requested operation. (0x8007139F)
  Source=VirtualDesktop
  StackTrace:
   at WindowsDesktop.Interop.ComWrapperBase`1.InvokeMethod[T](Object[] parameters, String methodName) in WindowsDesktop.Interop\ComWrapperBase.cs:line 68
   at WindowsDesktop.VirtualDesktop.EventProxy.ViewVirtualDesktopChanged(IApplicationView pView) in WindowsDesktop\VirtualDesktop.cs:line 58
   at WindowsDesktop.Interop.Build10240.VirtualDesktopNotification.ViewVirtualDesktopChanged(IApplicationView pView)

System.Runtime.InteropServices.COMException
  HResult=0x8007139F
  Message=The group or resource is not in the correct state to perform the requested operation. (0x8007139F)
  Source=VirtualDesktop
  StackTrace:
   at WindowsDesktop.Interop.ComWrapperBase`1.InvokeMethod[T](Object[] parameters, String methodName) in WindowsDesktop.Interop\ComWrapperBase.cs:line 68
   at WindowsDesktop.VirtualDesktop.EventProxy.ViewVirtualDesktopChanged(IApplicationView pView) in WindowsDesktop\VirtualDesktop.cs:line 58
   at WindowsDesktop.Interop.Build10240.VirtualDesktopNotification.ViewVirtualDesktopChanged(IApplicationView pView)

it's a COMException (so hopefully it can be caught):

rafo•5mo ago

another exception caused by the virtual desktop dll can be replicated by just clicking the "install" button here: https://apps.microsoft.com/detail/9pgjgd53tn86?launch=true&mode=mini&hl=en-us&gl=CZ

Microsoft Apps

WinDbg - Official app in the Microsoft Store

We've updated WinDbg to have more modern visuals, faster windows, a full-fledged scripting experience, and Time Travel Debugging, all with the easily extensible debugger data model front and center. WinDbg is using the same underlying engine as WinDbg (classic), so all the commands, extensions, and workflows you're used to will still work as the...

Amadeo•5mo ago

Wait The install button for unrelated app causes crash? Which version of windows do you have currently? We've had good share of issues with the VD library but I thought they solved them all I haven't changed anything with virtual desktop code/lib between the various audio test versions so something must have changed - still the install button causing crash is like wtf

rafo•5mo ago

Yup, it opens Microsoft Store in some weird, small-ish window. Windows 10, latest patches Weird thing is that I don't have any VD sensors enabled..

rafo•5mo ago

looks like this:

Amadeo•5mo ago

The VD library might have some internal thread/events going on that are then captured, information is stored for the consumers (us) to use I'll need to take a deeper look at it At least no more audio crashes... SO FAR 😄 just to be sure, the open/install in windows store is 100% reproducible?

Amadeo•5mo ago

did a sanity check we're using the newest version https://www.nuget.org/packages/Slions.VirtualDesktop/6.6.0#readme-body-tab 6.6.0

Slions.VirtualDesktop 6.6.0

C# Wrapper for the Virtual Desktop API on Windows 11 (and Windows 10).

Amadeo•5mo ago

@rafo in the meantime please: 1. close HASS.Agent 2. go to a) "C:\Users\<user>\AppData\Roaming\slions.net" b) "C:\Users\<user>\AppData\Local\slions.net" 3. remove everything you see there 4. start HASS.Agent I don't like the fact that your generated dll is 10240 and not 19045

Amadeo•5mo ago

for me with https://b.chihi.ro/hnzlcO.png

Amadeo•5mo ago

it's correct one (I also did the steps) https://b.chihi.ro/DJSCTh.png

rafo•5mo ago

done, I'm getting the same error:

Amadeo•5mo ago

what the actuall f now, this is a lot to ask but, could you please install the previous test version and see if it behaves the same? just to be sure if those are isolated or connected somehow

rafo•5mo ago

just to be sure, I'm not getting insane 😄

Amadeo•5mo ago

if I won't find anything with by current win version I'll update my PC and see then if I can reproduce it might even do that right now brb 😄

rafo•5mo ago

if you look at the call stack, the DLL that is being loaded is 19045, but it then loads WindowsDesktop.Interop.Build10240.ApplicationView

rafo•5mo ago

Amadeo•5mo ago

true noticed that on the second screenshot

rafo•5mo ago

installing this one (CSCore based): https://github.com/amadeo-alex/HASS.Agent/releases/tag/2.0.1-beta2-audio-sleep2

GitHub

Release 2.0.1-beta2-audio-sleep2 · amadeo-alex/HASS.Agent

Please do not download files from this "release" It's only purpose if to provide testing bed for the insiders/issue reporters. Unless explicitly instructed by the developers, please d...

Amadeo•5mo ago

https://b.chihi.ro/lL2OBk.png

rafo•5mo ago

call stack from the old version:

Amadeo•5mo ago

let's see now wait

rafo•5mo ago

that's old still

Amadeo•5mo ago

I still have older wtf windows

rafo•5mo ago

that's patchset from feb 2024 https://support.microsoft.com/en-au/topic/february-13-2024-kb5034763-os-builds-19044-4046-and-19045-4046-f1c993e4-32b3-4cc8-947a-69e70ae124d5

Amadeo•5mo ago

oh, now it wants me to reboot again?

rafo•5mo ago

you are quite behind, lol

Amadeo•5mo ago

I swear, if not for the HASS.Agent development I'd be doing "sudo apt upgrade" for a loooong time yeah, I have all autoupdates disabled 😄 I'm "that person" for which windows updates = evening spent investigating why something is not working

rafo•5mo ago

yah, I guess you'll get used to it. Just don't update on Friday 😄

Amadeo•5mo ago

Nah, Windows Education edition and GPOs are my friend 😄

rafo•5mo ago

aah, GPOs suck your friendly admin always means well 🙈

Amadeo•5mo ago

Haha xD I'm my own admin 😛 Thanks to them I have set of buttons in the start menu allowing me to update the os exactly when I want And not when m$shit thinks is appropriate

rafo•5mo ago

aah, I see

Amadeo•5mo ago

https://b.chihi.ro/ty63zj.png

Amadeo•5mo ago

that's better let's see

rafo•5mo ago

so, clicking the "install" button in the web browser should open a mini Microsoft Store window just like this one:

rafo•5mo ago

it throws the exception just after clicking the "install" button on the web (before this window even appears)

Amadeo•5mo ago

so two separate issues that's good I mean bad but also good

Amadeo•5mo ago

https://b.chihi.ro/kLv4lE.png

Amadeo•5mo ago

so now why it works for me but not for you and what clicking it has to do with virtual desktops

rafo•5mo ago

I guess it tries to call the getThumbnalWindow for every window that is opened? For some previews?

Amadeo•5mo ago

but the same should happen for me

rafo•5mo ago

how many Virtual desktops do you have? I have 2 2nd one is empty (no windows)

Amadeo•5mo ago

do you have any windows "customizations"? what browser are you using? 2

Amadeo•5mo ago

https://b.chihi.ro/Im4Kij.png

rafo•5mo ago

I don't exactly know what you mean firefox dev edition let me try edge

Amadeo•5mo ago

shut up windows or other "fix your privacy" stuff

rafo•5mo ago

I don't recall installing anything like that, but it's possible, the OS is installed since like 2020.. I do run dns-based adblock (something like a pi-hole), I imagine it blocks a lot of the phone home stuff.

Amadeo•5mo ago

that shouldn't affect anything on a second note, @barrelltitor would you be so kind to test this version also?

rafo•5mo ago

hmm, it's got something to do with hass settings

Amadeo•5mo ago

uuuu

rafo•5mo ago

aa, nope 😄

Amadeo•5mo ago

oh, message dissapeared 😄

rafo•5mo ago

changed the icon, it fooled me.. false alarm

Amadeo•5mo ago

wait so it did crash on other pc? or not

rafo•5mo ago

Amadeo•5mo ago

hmm this + the fact that I cannot reproduce it would suggest issue with the machine itself but now what

rafo•5mo ago

I can't reproduce it on my other PC too

Amadeo•5mo ago

does the crash also occur with Edge on the problematic pc?

rafo•5mo ago

yup could you test when you are attached to the hass.agent.exe in VS? It kinda looks like it's only VS that is sensitive, hass.agent doesn't actually crash (if VS debugger is not attached).. so when VS is not attached, it just logs something to the .log, but does not crash (it seems)

Amadeo•5mo ago

wait what what does it log? damnn, this is getting more and more interesting

rafo•5mo ago

message.txt

rafo•5mo ago

I don't actually think it's an issue, if it doesn't crash..

Amadeo•5mo ago

yeah, it's "FirstChanceException" so when debug logging is off it doesn't even show lemme test that and guess what I have in the logs

Amadeo•5mo ago

https://b.chihi.ro/yZNXLm.png

Amadeo•5mo ago

ignore the text being in kurwa language it means the same as your error

rafo•5mo ago

bardzo dobry 🙂

Amadeo•5mo ago

hmm

Amadeo•5mo ago

got this https://b.chihi.ro/WAWwaq.png

Amadeo•5mo ago

which means it throws but VS didn't catch that is there a chance that you have VS set to break on all exceptions? not only the unhandled ones

rafo•5mo ago

it's possible, any clue where should I look?

rafo•5mo ago

maybe the "just my code"?

Amadeo•5mo ago

https://b.chihi.ro/lfZbfA.png

Amadeo•5mo ago

check if you everything here selected https://b.chihi.ro/RsmvaH.png

Amadeo•5mo ago

I mean the boxes "under" that entry I have only one there that I enbaled way back oh no, more than one but most of them are unchecked

rafo•5mo ago

most of them are unchecked, yes

Amadeo•5mo ago

hmmm and those?

Amadeo•5mo ago

https://b.chihi.ro/K70zSX.png

rafo•5mo ago

Amadeo•5mo ago

wtf

rafo•5mo ago

I must have been toying with this, though. I vaguely remember looking into something.. I think it's this: break when exceptions occur cross AppDomain

Amadeo•5mo ago

lemme try

rafo•5mo ago

sure enough

Amadeo•5mo ago

rly? we've found it :D?

rafo•5mo ago

no crash in VS anymore *no exception in VS

barrelltitor•5mo ago

haven't had any leaks at all since, been sitting at a constant 60MB~ram usage since, or is it something else being tested now? haven't been keeping up i can help test anything that's needed, just need to know what to test

Amadeo•5mo ago

this version: https://github.com/amadeo-alex/HASS.Agent/releases/tag/2.0.1-alpha-audioleakfix-naudio rafo discovered a bug with CSCore that causes "unhandlable" exception when system sleeps/wakes up this version replaced CSCore with NAudio while keeping the event approach to audio handling

GitHub

Release 2.0.1-alpha-audioleakfix-naudio · amadeo-alex/HASS.Agent

Please do not download files from this "release" It's only purpose if to provide testing bed for the insiders/issue reporters. Unless explicitly instructed by the developers, please d...

barrelltitor•5mo ago

the installer says it failed to kill the satellite service, but when I try again it works fine

Amadeo•5mo ago

yeah, that's one bug I'm aware of but I don't think it's HASS.Agent issue more of a INNOSetup issue will see if I can remediate it with the reinstall rework

barrelltitor•5mo ago

Can't see any leaking or anything yet, ram is normal, around 100MB right now, will set it to sleep tonight and see if that causes any issues

DrR0x•5mo ago

this is going to have to go in our hall of fame for longest threads 🤣

rafo•5mo ago

It's been stable until now:

Faulting application name: HASS.Agent.exe, version: 2.1.0.0, time stamp: 0x661e0000
Faulting module name: Windows.Media.Playback.MediaPlayer.dll, version: 10.0.19041.4355, time stamp: 0xe86ba016
Exception code: 0xc0000005
Fault offset: 0x00000000000708a3
Faulting process id: 0xa9f4
Faulting application start time: 0x01daac436614db3c
Faulting application path: C:\Users\rafo\AppData\Local\HASS.Agent\Client\HASS.Agent.exe
Faulting module path: C:\Windows\System32\Windows.Media.Playback.MediaPlayer.dll
Report Id: 26237c3f-1bc9-454c-88f9-4e517ec9ff67
Faulting package full name: 
Faulting package-relative application ID:

Faulting application name: HASS.Agent.exe, version: 2.1.0.0, time stamp: 0x661e0000
Faulting module name: Windows.Media.Playback.MediaPlayer.dll, version: 10.0.19041.4355, time stamp: 0xe86ba016
Exception code: 0xc0000005
Fault offset: 0x00000000000708a3
Faulting process id: 0xa9f4
Faulting application start time: 0x01daac436614db3c
Faulting application path: C:\Users\rafo\AppData\Local\HASS.Agent\Client\HASS.Agent.exe
Faulting module path: C:\Windows\System32\Windows.Media.Playback.MediaPlayer.dll
Report Id: 26237c3f-1bc9-454c-88f9-4e517ec9ff67
Faulting package full name: 
Faulting package-relative application ID:

I've been playing no music, just sitting here in quiet, writing some emails?! Even VS did not catch this one, just crashed, above is from windows event log. is my RAM f*ed? ah, nope, it's the TTS again:

2024-05-23 19:17:06.326 +02:00 [INF] [MEDIA] Received media: http://192.168.112.219:8123/api/tts_proxy/ea6f7bd475eef055dfac4499188de60e5b8f7d3c_en_-_google_translate.mp3
2024-05-23 19:17:06.333 +02:00 [INF] [MEDIA] Playing: 20240523191706_99acee32.mp3

2024-05-23 19:17:06.326 +02:00 [INF] [MEDIA] Received media: http://192.168.112.219:8123/api/tts_proxy/ea6f7bd475eef055dfac4499188de60e5b8f7d3c_en_-_google_translate.mp3
2024-05-23 19:17:06.333 +02:00 [INF] [MEDIA] Playing: 20240523191706_99acee32.mp3

this was logged in hass.agent log at the exact same time as the crash so it was probably the TTS part, judging from the dll

Amadeo•5mo ago

now a windows system32 dll? something doesn't add up here

rafo•5mo ago

I did a sfc /scannow and chkdsk, let's see if it comes back

barrelltitor•5mo ago

any specifics on the way you did the TTS so I can test too?

rafo•5mo ago

I have an automation, where one of the actions is this:

- service: tts.google_translate_say
        data:
          cache: true
          entity_id: media_player.desktop
          message: slouch much?

- service: tts.google_translate_say
        data:
          cache: true
          entity_id: media_player.desktop
          message: slouch much?

It just says "slouch much?" through the media player (speakers) on my desktop PC

barrelltitor•5mo ago

hmm i dont have google translate setup might be related maybe? will try with others aah got it to crash @Amadeo repro steps: just spam a TTS until it happens

barrelltitor•5mo ago

this time it took longer but it still happened

barrelltitor•5mo ago

here's some logs, unredacted

2024-05-22_HASS.Agen...

2024-05-25_HASS.Agen...

Amadeo•5mo ago

Ohh nice Thanks for the video a d logs, I'll look into it

rafo•5mo ago

replicated the same thing on my side, also captured a crash dump, here's the output: https://pastebin.com/EeQJ7KhH

Pastebin

* Preparing the environment for Debugger Extensions Gal...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

rafo•5mo ago

here's analysis from windbg (stack trace): https://pastebin.com/XMLrQpYc

Pastebin

* Preparing the environment for Debugger Extensions Gal...

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

rafo•5mo ago

I've also got the .dmp file if needed..

rafo•5mo ago

I might be wrong, but the code crash seems to be happening somewere around here: https://github.com/tpn/winsdk-10/blob/master/Include/10.0.16299.0/winrt/windows.media.playback.idl#L1005

GitHub

winsdk-10/Include/10.0.16299.0/winrt/windows.media.playback.idl at ...

Contribute to tpn/winsdk-10 development by creating an account on GitHub.

Amadeo•5mo ago

I don't have access to my main PC for a few days so might be slow to respond but thank you for the research! I'll read through it this evening

rafo•5mo ago

I've noticed we're using a deprecated method here: https://github.com/amadeo-alex/HASS.Agent/blob/e08c93766c5ff482c0a70adf2a31564408c65b2f/src/HASS.Agent/HASS.Agent/Media/MediaManager.cs#L411C2-L411C72 - though this is most likely NOT the cause of the issue

GitHub

HASS.Agent/src/HASS.Agent/HASS.Agent/Media/MediaManager.cs at e08c9...

DEV PLATFORM - go to https://github.com/hass-agent/HASS.Agent for downloads :) - amadeo-alex/HASS.Agent

rafo•5mo ago

@barrelltitor are you able to replicate the error with the hass.agent window not being shown?

barrelltitor•5mo ago

good timing, just got out of the shower, checking now yteah just did

rafo•5mo ago

strange, It did not crash when my hass.agent windows was closed (only systray icon remains)

barrelltitor•5mo ago

the longer its been on the faster i can reproduce it it seems

barrelltitor•5mo ago

This is just after the earlier crash from testing, and I had to spam it for quite a bit but it did crash I open and close the GUI at the beginning of the video to make sure it's not open and I'm just blind. In the middle of the video I just hover over the icon to see if it crashed or not, sometimes the icon still stays

Amadeo•5mo ago

sooo vacation is over time to look back at the audio stuff 😄 lol replicated it just said "lol thx" and closed, visual studio didn't bother

The program '[27756] HASS.Agent.exe' has exited with code 3221225477 (0xc0000005) 'Access violation'.

❤️ and reminder to self: windows event viewer is not themed - nice way to flashbang yourself funny thing, event viewer doesn't have anything interesting to say about it now I need to get some kind of macro player going or else I'm going to kill my MMB clicker life expetancy 😄 I want to have at least 10 minutes of constant clicking untill marking the solution

Amadeo•5mo ago

so, 20 minutes of this and no crash https://b.chihi.ro/hVsFw4.png

Amadeo•5mo ago

I'll build it and give you a link

Amadeo•5mo ago

@sreknob and @barrelltitor if you would be so kind 🙂 https://github.com/amadeo-alex/HASS.Agent/releases/tag/2.1.0-alpha-audioleakfix-naudio

GitHub

Release 2.1.0-alpha-audiofix-naudio2 · amadeo-alex/HASS.Agent

Please do not download files from this "release" It's only purpose if to provide testing bed for the insiders/issue reporters. Unless explicitly instructed by the developers, please d...

sreknob•5mo ago

Thanks for the @ mention, I've been busy with other things and shamefully absent from this thread. Testing now!

barrelltitor•5mo ago

woah that was quick, wwwill get it tested tonight

Amadeo•5mo ago

what I did as a first try was to use non-deprecated api for playing the media lol surprisingly it did the trick for me

rafo•5mo ago

what a guy, installing right now, thx! wonder if they deprecated it for this exact reason 😄

Amadeo•5mo ago

forgot to also ping you, thank you for testing also 🙂

barrelltitor•5mo ago

Getting home and installing it. Is the way I tested it OK? I know you're not supposed to spam stuff a billion times every second, but I figured it shouldn't really error either way and the crash seemed similar

Amadeo•5mo ago

It's more than correct It shown me a way to replicate it 😄 It shouldn't crash even when spammed for 8 days - and even so with a normal exception amd not a a winrc5 and silent close

barrelltitor•5mo ago

seems fine now might setup an autoclicker and leave it for an hour to see what's what or leave it on while I sleep 👁️ But I haven't gone a full minute before without it crashing, so I'd say it's good, let's see if rafo still has the issue

Amadeo•5mo ago

❤️

rafo•5mo ago

been running it since sunday, it's been rock solid. Maybe I can also try the autoclicker thing, just to be sure (i'd appreciate a quick guide on how to set it up)

Amadeo•5mo ago

what kind of mouse you have :D?

rafo•5mo ago

https://www.trust.com/en/product/22991-gxt-144-rexx-ergonomic-vertical-gaming-mouse

Trust.com - GXT 144 Rexx Ergonomic Vertical Gaming Mouse

Vertical gaming mouse with ergonomic design to reduce arm and wrist strain

Amadeo•5mo ago

Hmm, does it come with software that has "macro" somewhere?

rafo•5mo ago

Yup, yup. I'll check it out. I'm also running a zmk-based keyboard so practically any macro can be programmed there.

vandelaybreaks•5mo ago

Satellite service isn't persisting after I log out. Any idea what I did wrong?

DrR0x•5mo ago

I'll answer in #🛰・general as it's irrelevant to this thread

Amadeo•5mo ago

can I be hopeful and treat the silence as a sign of completed fix :D?

sreknob•5mo ago

Yes - All has been well on my end. Thank you guys!

rafo•5mo ago

It's been a smoth ride, 0 crashes 🙂

Amadeo•5mo ago

I just noticed as part of other issue investigation that I forgot to include mute status of devices 😄 this to be specific

Muted = device.MMDevice.AudioEndpointVolume.Mute,

Muted = device.MMDevice.AudioEndpointVolume.Mute,

Ellie/jack5mikemotown•4mo ago

hey, dont want to make a new post if this is a continuing issue. but i can make one if necessary. i noticed what looked like unusually high cpu usage, so I opened the old hass agent too, and side by side, I'm noticing that resource monitor shows more than double, and sometimes even triple the cpu usage of the old hass agent. everything is almost completely identical between the two, no reason one should be using more resources than the other

Amadeo•4mo ago

In general this is solved with beta2 At least the audio memory issue What version is that?

Ellie/jack5mikemotown•4mo ago

2.0.1 Should I give the beta a shot?

DrR0x•4mo ago

Yep, the one in #🚀・releases 2.1.0-beta2 Couple more features as well as fixed for these issues

Gaming

Programming

High CPU and memory usage