engineertdog Posts - Answer Overflow

engineertdog

•Created by engineertdog on 7/31/2024 in #help

Parallel list

I'm trying to write to a list in parallel due to vendor SDK performance issues with sequential operations. I have the following code. Is this a proper approach?

c#
    private readonly ConcurrentQueue<DRT> _recordQueue = new ConcurrentQueue<DRT>();
private readonly SemaphoreSlim _flushLock = new SemaphoreSlim(1, 1);

    public async Task Execute() {
      try {
        // external/irrelevant to the list question
        await PerformWorkerSetup();
        IEnumerable<SRT> sourceRecords = ExtractRecords();
        //

        await TransformRecords(sourceRecords);
        await FlushRecords(1);
      } catch (Exception ex) {
        Log.Error(ex.Message);
      }
    }

    public async Task TransformRecordsParallel(IEnumerable<SRT> sourceRecords) {
      IEnumerable<Task> tasks = sourceRecords.Select(async (record) => {
        _recordQueue.Enqueue(this.FormatRecord(record));
        await FlushRecords(Config.Threshold);
      });

      await Task.WhenAll(tasks);
    }

    public async Task FlushRecords(int threshold) {
      if (_recordQueue.Count >= threshold) {
        await _flushLock.WaitAsync();

        try {
          IList<DRT> recordList = new List<DRT>();

          // Pull records out of the bag for processing.
          while (recordList.Count <= threshold && _recordQueue.TryDequeue(out DRT record)) {
            recordList.Add(record);
          }

          // write the recordList to file or SQL
        } catch {
          throw;
        } finally {
          _flushLock.Release();
        }
      }
    }

c#
    private readonly ConcurrentQueue<DRT> _recordQueue = new ConcurrentQueue<DRT>();
private readonly SemaphoreSlim _flushLock = new SemaphoreSlim(1, 1);

    public async Task Execute() {
      try {
        // external/irrelevant to the list question
        await PerformWorkerSetup();
        IEnumerable<SRT> sourceRecords = ExtractRecords();
        //

        await TransformRecords(sourceRecords);
        await FlushRecords(1);
      } catch (Exception ex) {
        Log.Error(ex.Message);
      }
    }

    public async Task TransformRecordsParallel(IEnumerable<SRT> sourceRecords) {
      IEnumerable<Task> tasks = sourceRecords.Select(async (record) => {
        _recordQueue.Enqueue(this.FormatRecord(record));
        await FlushRecords(Config.Threshold);
      });

      await Task.WhenAll(tasks);
    }

    public async Task FlushRecords(int threshold) {
      if (_recordQueue.Count >= threshold) {
        await _flushLock.WaitAsync();

        try {
          IList<DRT> recordList = new List<DRT>();

          // Pull records out of the bag for processing.
          while (recordList.Count <= threshold && _recordQueue.TryDequeue(out DRT record)) {
            recordList.Add(record);
          }

          // write the recordList to file or SQL
        } catch {
          throw;
        } finally {
          _flushLock.Release();
        }
      }
    }

4 replies

CC#

•Created by engineertdog on 7/30/2024 in #help

Base Class architecture

I'm working on building a highly versatile base class that I intend to use in numerous applications and projects, so I'm trying to theorize the best way to build this class (see code attached). Requirements: - Quartz is the scheduling mechanism that will be used in the apps (and the base worker should be derived from that structure) - In the UIs, users will be able to choose workflows (which are tied to specific applications) and be able to choose the source/destination (ie SDK->SQL or SQL->File, etc) - The rest of configuration (stored in SQL) will be written to file, somewhere, that the apps will pick up through this Quartz worker. For example: - A user goes into the UI and selects that they want to pull connection statistics from Server A ("connection statistics" would be tied to a specific class that would inherit from the base, which defines the powershell/sdk/etc to extract or flush records in the correct format) - The user chooses the server to pull the statistics from. - The user chooses to write the output to file. - The user chooses to execute the task now. Or another example: - A user goes into the UI and selects that they want to write data to Server B - The user chooses the server to write data to - The user chooses SQL as the source to select xx records - The user chooses to write the output to the server itself (ie via SDK)

4 replies

CC#

•Created by engineertdog on 7/11/2024 in #help

Multipart form upload prepends the file

I have a 3rd party that I am uploading files to via C#. For some reason, it looks like C# is prepending the files with additional information prior to uploading them and I can't figure out where it is occurring. Example prepend to CSV or JSON files

--<file-hash>
Content-Disposition: form-data; name=file; filename=<file-name>;
filename*=utf-8''<file-name>
{
...
}

--<file-hash>
Content-Disposition: form-data; name=file; filename=<file-name>;
filename*=utf-8''<file-name>
{
...
}

c#
HttpClient client = new();
MutlipartFormDataContent formData = [];
FileStream fileStream = new("filepath", FileMode.Open, FileAccess.Read);
StreamContent fileContent = new(fileStream);
formData.Add(fileContent, "file", "filename.csv");
client.PutAsync("url", formData);

c#
HttpClient client = new();
MutlipartFormDataContent formData = [];
FileStream fileStream = new("filepath", FileMode.Open, FileAccess.Read);
StreamContent fileContent = new(fileStream);
formData.Add(fileContent, "file", "filename.csv");
client.PutAsync("url", formData);

1 replies

CC#

•Created by engineertdog on 6/27/2024 in #help

Update configuration in memory

I have the class below where I'm trying to provide default values if a user doesn't configure them in appsettings. Supposedly, this is supposed to work, but it's not working in practices. Do you have any recommendations to either fix this, or to just use default values regardless and screw the config?

c#
  public static class MainelyLog {
    private readonly static IConfigurationRoot _configuration;
    private readonly static string _appDirectory = AppDomain.CurrentDomain.BaseDirectory;

    static MainelyLog() {
      _configuration = new ConfigurationBuilder()
        .SetBasePath(_appDirectory)
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .Build();
    }

    public static void SetupLogging(string defaultLogPath) {
      IConfigurationSection? logConfig = _configuration.GetSection("Serilog:WriteTo")
        .GetChildren()
        .FirstOrDefault(c => c["Name"] == "File");

      if (logConfig != null && !Path.IsPathFullyQualified(logConfig["Args:path"])) {
        string logPath = logConfig["Args:path"] ?? defaultLogPath;
        if (logPath.StartsWith(".\\")) { logPath = logPath[2..]; }
        logConfig["Args:path"] = Path.Combine(_appDirectory, logPath);
      }

      Log.Logger = new LoggerConfiguration()
      .ReadFrom.Configuration(_configuration)
      .CreateLogger();
    }
  }

c#
  public static class MainelyLog {
    private readonly static IConfigurationRoot _configuration;
    private readonly static string _appDirectory = AppDomain.CurrentDomain.BaseDirectory;

    static MainelyLog() {
      _configuration = new ConfigurationBuilder()
        .SetBasePath(_appDirectory)
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .Build();
    }

    public static void SetupLogging(string defaultLogPath) {
      IConfigurationSection? logConfig = _configuration.GetSection("Serilog:WriteTo")
        .GetChildren()
        .FirstOrDefault(c => c["Name"] == "File");

      if (logConfig != null && !Path.IsPathFullyQualified(logConfig["Args:path"])) {
        string logPath = logConfig["Args:path"] ?? defaultLogPath;
        if (logPath.StartsWith(".\\")) { logPath = logPath[2..]; }
        logConfig["Args:path"] = Path.Combine(_appDirectory, logPath);
      }

      Log.Logger = new LoggerConfiguration()
      .ReadFrom.Configuration(_configuration)
      .CreateLogger();
    }
  }

The purpose of doing this is to account for relative paths not being correct when running interactively vs as a service.

80 replies

CC#

•Created by engineertdog on 6/26/2024 in #help

Uncaught exception

I have the following code as part of a class I'm building for impersonation. It works, but the UnauthorizedAccessException is being marked as unhandled. If I hit continue, it all works as expected. But why is it unhandled and how can I address that?

c#
try {
  WindowsIdentity.RunImpersonated(_safeAccessTokenHandle, () => {
    // This will throw if the impersonation fails
    //string impersonatedUser = WindowsIdentity.GetCurrent().Name;
    //Console.WriteLine("Impersonation successful, current user: " + impersonatedUser);

    try {
      string securePath = "";
      if (!Directory.Exists(securePath)) {
        throw new UnauthorizedAccessException($"Access to {securePath} was denied.");
      }
    } catch {
      throw;
    }
  });
} catch (Exception ex) {
  Console.WriteLine($"Impersonation validation failed: {ex.Message}");
  Dispose();
  // Re-throw to ensure the exception is handled by the caller
  throw;
}

c#
try {
  WindowsIdentity.RunImpersonated(_safeAccessTokenHandle, () => {
    // This will throw if the impersonation fails
    //string impersonatedUser = WindowsIdentity.GetCurrent().Name;
    //Console.WriteLine("Impersonation successful, current user: " + impersonatedUser);

    try {
      string securePath = "";
      if (!Directory.Exists(securePath)) {
        throw new UnauthorizedAccessException($"Access to {securePath} was denied.");
      }
    } catch {
      throw;
    }
  });
} catch (Exception ex) {
  Console.WriteLine($"Impersonation validation failed: {ex.Message}");
  Dispose();
  // Re-throw to ensure the exception is handled by the caller
  throw;
}

13 replies

CC#

•Created by engineertdog on 6/25/2024 in #help

✅ .net8 impersonate

c#
bool returnValue = LogonUser("", "", "",
    LOGON32_LOGON_BATCH, LOGON32_PROVIDER_DEFAULT,
    out safeAccessTokenHandle);

WindowsIdentity.RunImpersonated(
 safeAccessTokenHandle,
 // User action
 () => {
   // Check the identity.
   Console.WriteLine("During impersonation: " + WindowsIdentity.GetCurrent().Name);
 }
 );

c#
bool returnValue = LogonUser("", "", "",
    LOGON32_LOGON_BATCH, LOGON32_PROVIDER_DEFAULT,
    out safeAccessTokenHandle);

WindowsIdentity.RunImpersonated(
 safeAccessTokenHandle,
 // User action
 () => {
   // Check the identity.
   Console.WriteLine("During impersonation: " + WindowsIdentity.GetCurrent().Name);
 }
 );

This logs the current user and not the identity that's impersonated. I need to be able to execute certain tasks as users. Is this just not reasonable with net8? Nevermind, the current identity lied. That was code taken from the MS docs website. If I do an action, then it does use that context. Misleading that the current identity isn't correct.

1 replies

CC#

•Created by engineertdog on 6/24/2024 in #help

✅ Content would exceed Content-Length

I have this code to add files to an HttpClient to send API requests to an external API.

c#
    public void AddFile(string filePath, string fileName) {
      // Open the file stream
      FileStream fileStream = new(filePath, FileMode.Open, FileAccess.Read);
      _fileStreams.Add(fileStream);

      // Create stream content and add it to the form data.
      StreamContent streamContent = new(fileStream);
      streamContent.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
      _formData.Add(streamContent, "file", fileName);
    }

c#
    public void AddFile(string filePath, string fileName) {
      // Open the file stream
      FileStream fileStream = new(filePath, FileMode.Open, FileAccess.Read);
      _fileStreams.Add(fileStream);

      // Create stream content and add it to the form data.
      StreamContent streamContent = new(fileStream);
      streamContent.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
      _formData.Add(streamContent, "file", fileName);
    }

The problem I have, is that I am getting the error Unable to write content to request stream; content would exceed Content-Length. Now, in some cases, the automated Content-Length value being set was incorrect. I have tried manually setting the content length with streamContent.Headers.ContentLength = fileStream.Length and while this does correctly set the size, I still get the same error that it would exceed the length. I'm not sure why that's occurring.

3 replies

CC#

•Created by engineertdog on 2/24/2024 in #help

EF6 Relationships

I have two classes built in EF Core, that I used to scaffold the database and use in the frontend Blazor app.

c#
  public class StreamPointListenerStep {
    public Guid Id { get; set; } = Guid.NewGuid();
    //....
    public StreamPointIntegrationELogger? StreamPointIntegrationELogger { get; set; } = null;
  }

  public class StreamPointIntegrationELogger {
    public Guid Id { get; set; } = Guid.NewGuid();

    [ForeignKey(nameof(StreamPointListenerStep))]
    public Guid StepId { get; set; }
    public StreamPointListenerStep? StreamPointListenerStep { get; set; } = null;
    //...
  }

c#
  public class StreamPointListenerStep {
    public Guid Id { get; set; } = Guid.NewGuid();
    //....
    public StreamPointIntegrationELogger? StreamPointIntegrationELogger { get; set; } = null;
  }

  public class StreamPointIntegrationELogger {
    public Guid Id { get; set; } = Guid.NewGuid();

    [ForeignKey(nameof(StreamPointListenerStep))]
    public Guid StepId { get; set; }
    public StreamPointListenerStep? StreamPointListenerStep { get; set; } = null;
    //...
  }

Now, building and using this in Blazor is fine. However, I also need the table in a .NET Framework app. So I've ported the DB model to EF6 just to simply use the context for accessing the database and the models without writing the queries manually. However, EF6 refuses to work with this due to the relationship. It throws an error for

Because the Dependent Role properties are not the key properties, the upper bound of the multiplicity of the Dependent Role must be '*'.

What solutions are there for getting around this? I have to use .NET Framework, and building an API is overhead that's just not needed.

115 replies

CC#

•Created by engineertdog on 2/23/2024 in #help

C# 8 logging

6 replies

CC#

•Created by engineertdog on 2/22/2024 in #help

Best way to model different options in EF Core

I have a class that's used to define the integration parameters for a particular process. Now, I have integration options A,B,C, etc each of which correspond to different tables with different IDs I need to populate on my class. ex

c#
public class myclass {
    public string MyParam1 { get; set; }
    public string MyParam2 { get; set; }
    public string MyParam3 { get; set; }

    // Integration A
    public Guid PackageId { get; set; }
    public Guid LocationId { get; set; }
    public Guid ScreenElementId { get; set; }
    // Integration B
    public Guid PackageId { get; set; }
    public Guid LocationId { get; set; }
    public Guid ScreenElementId { get; set; }
    // etc
}

c#
public class myclass {
    public string MyParam1 { get; set; }
    public string MyParam2 { get; set; }
    public string MyParam3 { get; set; }

    // Integration A
    public Guid PackageId { get; set; }
    public Guid LocationId { get; set; }
    public Guid ScreenElementId { get; set; }
    // Integration B
    public Guid PackageId { get; set; }
    public Guid LocationId { get; set; }
    public Guid ScreenElementId { get; set; }
    // etc
}

Now, only 1 integration is supported for each instance of myclass. I could in theory define each available integration's FK references in this table, but there must be a better way to do this?

c#
public class myclass {
    public string MyParam1 { get; set; }
    public string MyParam2 { get; set; }
    public string MyParam3 { get; set; }

    // Integration
    public IntegrationType Integration { get; set; } // A,B,C,etc
    public Guid IntegrationId { get; set; } // But wouldn't this just then be a table with all references, like in the previous example?
}

c#
public class myclass {
    public string MyParam1 { get; set; }
    public string MyParam2 { get; set; }
    public string MyParam3 { get; set; }

    // Integration
    public IntegrationType Integration { get; set; } // A,B,C,etc
    public Guid IntegrationId { get; set; } // But wouldn't this just then be a table with all references, like in the previous example?
}

5 replies

CC#

•Created by engineertdog on 2/8/2024 in #help

Variable type passed to generic

Is there any way to get around this being an error?

c#
Type type = Type.GetType("OSIsoft.PI.Net, PIThreadInfo");

return powershell.Invoke<type>()

c#
Type type = Type.GetType("OSIsoft.PI.Net, PIThreadInfo");

return powershell.Invoke<type>()

33 replies

CC#

•Created by engineertdog on 2/7/2024 in #help

C# executed in NodeJS

I have a 3rd party vendor SDK that's written in .NET Framework. They're working on a NET Core version, but I'd still like to look at the possible usage with NodeJS. Are there any performance or other issues to worry about? The main reason I look to use the vendor's SDK instead of their API is performance between the technologies they offer, where the API is based on the SDK and is much slower to use.

10 replies

CC#

•Created by engineertdog on 2/5/2024 in #help

.HasPrincipalKey -> Specify different column name

I need to use .HasPrincipalKey in Fluent, but I need the resulting column to be named differently from what the Fluent API wants to use. I don't see an option to override the column name to something different?

5 replies

CC#

•Created by engineertdog on 2/5/2024 in #help

FK on Guid

I have the following two classes

c#
  public class PiCollectiveIdentity {
    public Guid Id { get; set; } = new Guid();
    public long IdentityId { get; set; }
    public required string Name { get; set; }
    [ForeignKey("IdentityName")]
    public IList<PiCollectiveIdentityMapping> Mappings { get; set; } = new List<PiCollectiveIdentityMapping>();
  }

  public class PiCollectiveIdentityMapping{
    public Guid Id { get; set; } = new Guid();
    public DateTime CollectionDate { get; set; } = DateTime.UtcNow;
    public required string IdentityName { get; set; }
    public PiCollectiveIdentity ? Identity { get; set; } = null;
  }

c#
  public class PiCollectiveIdentity {
    public Guid Id { get; set; } = new Guid();
    public long IdentityId { get; set; }
    public required string Name { get; set; }
    [ForeignKey("IdentityName")]
    public IList<PiCollectiveIdentityMapping> Mappings { get; set; } = new List<PiCollectiveIdentityMapping>();
  }

  public class PiCollectiveIdentityMapping{
    public Guid Id { get; set; } = new Guid();
    public DateTime CollectionDate { get; set; } = DateTime.UtcNow;
    public required string IdentityName { get; set; }
    public PiCollectiveIdentity ? Identity { get; set; } = null;
  }

Where each database record will have it's own unique ID, but the relationship between the two records is based on name (due to pulling the data from a 3rd party system). However, this throws an error when attempting to migrate for

'PiCollectiveIdentityMapping.Mappings' with foreign key properties {'IdentityName' : string} cannot target the primary key {'Id' : Guid} because it is not compatible

What is the proper way to build the relationship?

7 replies

CC#

•Created by engineertdog on 12/21/2023 in #help

Persist data between Quartz Execution

I need to be able to persist data between instance executions. I have to tie two separate jobs together through a common identifier. The approach below does not work. From my understanding, I will need to create a new schedule based off the old one in order to persist data?

c#
public virtual async Task Execute(IJobExecutionContext context) {
  JobDataMap dataMap = context.MergedJobDataMap;
  long count = dataMap.GetLong("jobIterationNumber");
  count++;
  dataMap.Put("jobIterationNumber", count);
  Execute();

  // Instead, build a new schedule based on the old one?
  var oldMap = oldTrigger.JobDataMap;
  oldMap["jobIterationNumber"] = count;

  var oldTrigger = context.Trigger;
  var newTrigger = TriggerBuilder.Create()
    .WithIdentity(oldTrigger.Key.Name, oldTrigger.Key.Group)
    .UsingJobData(oldTrigger.JobDataMap)
    .WithSchedule(oldTrigger.GetScheduleBuilder())
    .Build();
  await context.Scheduler.ScheduleJob(newTrigger);
}

c#
public virtual async Task Execute(IJobExecutionContext context) {
  JobDataMap dataMap = context.MergedJobDataMap;
  long count = dataMap.GetLong("jobIterationNumber");
  count++;
  dataMap.Put("jobIterationNumber", count);
  Execute();

  // Instead, build a new schedule based on the old one?
  var oldMap = oldTrigger.JobDataMap;
  oldMap["jobIterationNumber"] = count;

  var oldTrigger = context.Trigger;
  var newTrigger = TriggerBuilder.Create()
    .WithIdentity(oldTrigger.Key.Name, oldTrigger.Key.Group)
    .UsingJobData(oldTrigger.JobDataMap)
    .WithSchedule(oldTrigger.GetScheduleBuilder())
    .Build();
  await context.Scheduler.ScheduleJob(newTrigger);
}

1 replies

CC#

•Created by engineertdog on 12/15/2023 in #help

✅ Convert 3rd party Dictionary to usable variable

4 replies

CC#

•Created by engineertdog on 12/14/2023 in #help

SqlBulkCopy IList

What frameworks are available for performing bulk copy using an IList as the input? There's DataTable and FastMember. I have used FastMember, but it's outdated and I'd rather use something current.

5 replies

CC#

•Created by engineertdog on 12/13/2023 in #help

What version of PowerShell.Create() is created?

PowerShell ps = PowerShell.Create().AddCommand("$PSVersionTable.PSVersion");
ps.Invoke();

PowerShell ps = PowerShell.Create().AddCommand("$PSVersionTable.PSVersion");
ps.Invoke();

This throws an error, stating that $PSVersionTable is not a function. I have PowerShell 7 installed where the code is running, and I've verified the commands still work there. So what is the SDK creating behind the scenes that's not compatible?

24 replies

CC#

•Created by engineertdog on 12/12/2023 in #help

Standard library usage problem

4 replies

CC#

•Created by engineertdog on 12/12/2023 in #help

Send message from one app to another

What's the easiest method for sharing data between applications? MessageQueue looked promising, but it doesn't appear available in Core. I need to be able to send data between different applications that are using both Core & Framework. So, Core<->Core, Core<->Framework communication. Can't use cloud services, although RabbitMQ could be used if there's nothing well supported natively or current on Nuget.

14 replies

Gaming

Programming