Kale Vivi Comments - Answer Overflow

Kale Vivi

•Created by Kale Vivi on 1/19/2025 in #java-help

Dealing with concurrent updates

How does the db or language know the order of the inserts/updates/changes? Is it based on ids or timestamps or..? I'll start reading up on the concepts above. thanks!

12 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

So in the case of async vs completable future. If I need info, I go with CF, otherwise, just use async?

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

One thing I've been thinking about though are different approaches to handling batch updates of large data sizes. There's things like @async, completable futures, spring batch, parallel streams, ... What are some considerations I should think about in understanding when to use what?

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

Hey sorry to respond late. I was able to resolve it by adding hibernate props for batch and the @async annotation earlier. That seemed to work wonders for perf. Thanks all!

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

I just want it done in 30 to 45 mins if possible?

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

It's still taking hours and exception is thrown with db io error. Db timeouts I guess before process can finish.

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

Hey Puggy, while it is a dramatic improvement over the previous implementation, it is still slow. (at least when running the job locally). Other approaches where would maybe include leveraging @Async and ThreadExecutor or parallel streams? Does that seem like the right approach forward?

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

Thank you, giving it a try now. You're right the for each loop code was slow

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

Not sure I follow the last part here. Let me share the builders to confirm understanding: One entity:

public static One fromPerson(Person person) {
    return One.builder()
            .idNumber(person.getIdNumber())
            .build();
}

Six entity:

public static List<Six> fromPerson(Person person) {
        return person.getSix().stream().map(s -> Six.builder()
                .idNumber(person.getIdNumber())
                .someField(s.getSomeField())
                .build()).collect(Collectors.toList());
    }
}

Should I just do saveAll for all and not do a loop or do a loop for entities like Six here? Thanks again

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

Thanks a lot Puggy! I got distracted by writing some tests but will take a look at the above in a minute and give it a try. Thank you. Just adding here in case the above wasn't clear for the savePeople(). I have a a list of Person objects where Person has single fields then under Person is a nested List of other objects (Person then for Person object there's a List<Six> six), so thats' why I had to use the saveAll for the other repos but not the first several where there's only one field/value for that person versus the others where person can have multiple records of that (e.g. Six here). Sorry for the names (just masked them, they're bad).

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

I think the issue is the savePeople above where I go one by one here. In repo fromPerson, that's where I take that domain object and map it to entity for each field. Should I just parallel streams there? It's crazy that I have to break down the one object above into smaller objects to persist in multiple tables too which is also what I feel impacts performance

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

@PersistenceContext
private final EntityManager entityManager;
private final RepoOne repoOne;
private final RepoTwo repoTwo;
private final RepoThree repoThree;
private final RepoFour repoFour;
private final RepoFive repoFive;
private final RepoSix repoSix;
private final RepoSeven repoSeven;
private final RepoEight repoEight;
private final RepoNine repoNine;

private static final int BATCH_SIZE = 1000;

@Override
@Transactional
public void saveAll(Set<Person> persons) {
 Iterable<List<Person>> peopleBatches = ListUtils.partition(persons.stream().toList(), BATCH_SIZE);
 List<Person> newPeople = new ArrayList<>();
 List<Person> existingPeople = new ArrayList<>();
 AtomicInteger indexPersonBatches = new AtomicInteger(1);

 for (List<Person> batch : peopleBatches) {
    for (Person person : batch) {
        if (existsBy(person.getIdNumber())) {
            existingPeople.add(person);
        } else {
            newPeople.add(person);
        }
    }   
        updatePeople(existingPeople);
        savePeople(newPeople);
        existingPeople.clear();
        newPeople.clear();
    }
    entityManager.flush();
    entityManager.clear();
}

private void savePeople(List<Person> newPeople) {
    for (Person person : newPeople) {
        repoOne.save(One.fromPerson(person));
        repoTwo.save(Two.fromPerson(person));
        repoThree.save(Three.fromPerson(person));
        repoFour.save(Four.fromPerson(person));
        repoFive.saveAll(Five.fromPerson(person));
        repoSix.saveAll(Six.fromPerson(person));
        repoSeven.saveAll(Seven.fromPerson(person));
        repoEight.saveAll(Eight.fromPerson(person));
        repoNine.saveAll(Nine.fromPerson(person));
    }
}

public boolean existsBy(String number) {
 return repoOne.findByIdNumber(number) != null;
}

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

Ok let me take a look there. Here's kind of what I have so far and I'm feeling like I'm definitely going to need to refactor this

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

My other concern is I'm also going one by one in the list to convert the domain object to entity before persistence and I'm worried about that also slowing performance

56 replies

JCHJava Community | Help. Code. Learn.

•Created by Kale Vivi on 12/16/2024 in #java-help

Sequence allocation size question

Thanks, I adjusted first to 500 and saw a good improvement. Trying 1000 and 1000 batch size now to see if it's even better

56 replies

JCHJava Community | Help. Code. Learn.

•Created by noreva on 12/14/2024 in #java-help

run in docker container error

It can't autowire the websecurityconfig. Did you take a look at that part of the code?

28 replies

Gaming

Programming