Official Fedora image build times:
Official Fedora image build times:
Silverblue - 8:29 AM UTC (latest)
Kinoite - 7:58 AM UTC (latest)
Sericea - 8:29 AM UTC (latest)
We build main at 7 AM UTC. @bsherman doesn't this mean we only need to push things back 2 hours?
147 Replies
let's double and triple check this 😄
Yeah... On second though these times look inline with MST for me. Rawhide says 13 hours ago, tag says it was pushed at 7:01 AM. It's 8 PM here
So assuming those are right - that means 2:29 PM UTC is the latest an image is pushed. So we'd build main at 3 PM UTC
so, i'm looking at the tag history tab on quay...
sericea, at least as of now, is always building last of those 3, so lets ignore the others and just look at it
https://quay.io/repository/fedora/fedora-sericea?tab=history
Quay
Quay is the best place to build, store, and distribute your containers. Public repositories are always free.
here we can see when the
39 tag was applied to the image each day, which is the last stage of image publishing
going back to Oct 13... the time varies, but the latest is 9:29 AMCST. Mine says 8:29
ok, good, we are looking at the same thing and now we know the webpage shows viewer's local time
So it is the current TZ set by the system and not UTC
yep
so, i'm currently UTC-5 ... so that 9:29AM is really 14:29
2:29PM
Yep
ok, so for purposes of this investigation, you good with moving to 24 hour time and UTC only?
Yeah
Working in our current TZs won't get us anywhere lol
cool, so our latest known build time is 14:29 for official upstream
and... the QA builds, fedora-ostree-desktops, are pretty much 12 hours before/after that
Yeah. It's like 3 or 4 AM UTC
so which is "first" 😄
Heh, that is the question
so lets look at these
https://quay.io/organization/fedora-ostree-desktops
Latest I see is Onyx at 4:10 AM
yea, but... "yesterday"
Oh yup true
Sericea at 2:22 AM, but official images cover that
Base@2:14
ok, i'm switching to local time for a moment.
lets just talk kinoite:
fedora/fedora-silverblue, built at 7:10AM for you today
fedora-ostree-desktops/kinoite, build at 7:47PM for you today
right?
Yep
ok... so that's
official kinoite at 1310
qa kinoite at 0147 (tomorrow)
right?
Yeah
to me, this says we need to pick a target... either relatively close to the official builds, or closer to the QA builds... and hope for the best
and of course, I'd push for the offiicial
so, if we say, 1430 is our current latest known official build time
... how far beyond that do we want to wait? 1 hour 1.5 hours?
start with 1 and see what happens?
Do we need an hour since 2:29 has been the absolute worst case so far? Would shooting for 30 minutes after work?
oh crap
all our config/akmods stuff builds against
base no?Let me check
config is fedora:latest... it doesn't care
FROM registry.fedoraproject.org/fedora:latest AS builder
Yyp
but akmods is building off fedora-ostree-desktops/base
i think that needs to change to match official
i mean, i'm not sure what happens if official silverblue/kinoite/sericea don't match
Build from official Silverblue?
Ope yup
but yeah, i would expect it more likely the official images match each other than they match
baseAgreed. We'll just have to find out and see
If things blow up, blame NVIDIA
i'm pulling all 3 of those images to check a few things quick
i imagine sericea will be smallest, but want to see
we could use that as our "base" for akmods maybe
oh! they do seem to share some layers, so that's cool
That's awesome
I would assume they share most of their layers?
os, yeah, sericea is smallest, not by much, so, i say we use it or silverblue to build akmods
silverblue is the first to build daily, so i dunno if that makes it most official or what
<--- wishes fedora was just built off coreos. 😦
well, it wouldn't be 38/39/40 then 😉 it'd jsut be stable/testing 😉
Since Sericea is smaller, I'd go with that
All of this makes me wish build hooks were a thing
ok, so ... i think means the changes to our core repos will be as follows:
config) change cron to daily at 1430 same time as latest known official build
akmods) change cron to daily at 1500 30 minutes after AND change it's FROM to quay.io/fedora/fedora-sericea instead of fedora-ostree-desktops/base
main) change cron to daily at 1530 i can add this to the issue if we agree and then we'll need to change all the others "joy"
main) change cron to daily at 1530 i can add this to the issue if we agree and then we'll need to change all the others "joy"
I was about to say, hwcompat, bazzite, and bluefin will need to be pushed back as well
main may need to wait more than 30 minutes for akmods
we have definitely seen it take over 20 minutes sometimes
Maybe build akmods at 3 then
main at 4
actually... we could streamline this more
1) config doesn't really care about silverblue/sericiea etc... since it is just a builder... for building our own RPMs so we can build it anytime... and it's always fast... like < 5minutes
2) akmods can build once upstream silverblue is ready, we don't need to wait for kinoite for that...
3) only main needs to wait for the full set of upstream images
Ah, yeah, that's true
akmods could build as early as like 1:30
but main does need config + akmods, and akmods does need config
*PM
1330
😉
I use 24 hour clocks on everything. Why I still bother with AM/PM, I have no idea lol
I started learning 24 hour clocks in high school French class... but the Army finished the job 😉
🙂
My brother got me in the habit because I always thought it was weird for clocks to just reset midday
i would say 1330 for config, 1400 for akmods (bult on silverblue), 1500 for main
Sounds great to me!
Absoutely perfect
and then all the trickle down cron fallout
1530-1600 for the hwe layers and then 1630 for bazzifin?
i don't know how fast those build
1530 NVIDIA
1600 hwcompat
Or rather
1530 NVIDIA, 1545 hwcompat
ik
too late lol
hackmd fellas
hackmd discord
i'm starting hackmd now
I'll PR CI cron changes
(again)
i'll do akmods/config/main at least
hackmd is being wonky for me
yeah it's being slow
hwcompat no longer relies on nvidia repo builds, correct?
Correct
so nvidia + hwcompat can all run same time... but... i think there's asus/surface nvidia sub builds
ugh
yes, this needs to be triggered somehow instead of cron'd
sec, I have an idea
asus/surface/and whatever the other one is can run at the same time NVIDIA runs in that case then hwcompat NVIDIA can run concurrently if we do that
But I'm thinking... these have over 40 builders each
We have 60 builders
ah
Better to space them no?
how's that?
How's what?
the edits in the doc
what doc?
Oh lol
I am putting what you're saying in here
twiddling my thumbs
what about surface-nvidia and asus-nvidia?
HACKMD LOADED!
1600 asus-nvidia
surface-nvidia
15 minutes is plenty for NVIDIA images
Highest I see anymore is 11 minutes
15 is not enough for surface/asus though, they hit 15 sometimes
If they're running concurrently
do akmods need an hour?
I just fixed that
checking, but no, i think 45 minutes is enough
oh
Then config could run at 1400
I mean let's leave enough slack, we are on a free(ish) tier
config only takes 5 minutes I thought?
but our 1500 time for main is not about waiting on akmods, its about waiting for official images
ah ok
1330 / 1400 /1500 for config/akmods/main should not be changed, those are related tp upstream
That's true so akmods could keep 1330
And still be good
so 1400 for akmods is fine since we can't do main earlier thatn 1500
That works too
1500 for main is the earliest it can go because of official image times
so, just work forward on our downstreams, don't need to change config / akmods
nod
What we have on hackmd looks good to me. If everyone here is in agreement, I'll go PR it all
current state looks good to me.
@EyeCantCU I'm happy to snag config/akmods/main
Sounds good. I pushed config already
how many builders does nvidia fire up?
LOL
By push I mean to a different branch
oh you know what
that's 11am my time
I'll keep the builder screen open on my screen
Let me look... I think it's 44
and then we'll watch to see if there's any adjustments to make
42
like I would expect the hwe stack ones to basically just queue up, but they'd be in order, so that's fine?
The answer to life, the universe, and everything
oh and we're dropping 37 soon
I think that's okay
so we should be good
Definitely
since we're basically worst case as it is
and then the unofficial images we can do later or whatever
I'll keep an eye on the review queue, thanks!
ugh, i've got to replicate the conditional version logic into akmods 😄
it's ok we maxed out the builders so you've got time lol
@EyeCantCU want to look at this one ?
https://github.com/ublue-os/akmods/pull/85
GitHub
feat: use official upstream images for kmod builds by bsherman · Pu...
This change does two things:
uses official upstream silverblue image, rather than the QA build of base, as our builder
changes cron to 1400 UTC to more closely match the time of official image bui...
since it's the logic changes, not just cron
I'll have a look
let's blog it tomorrow!
Looks fantastic
@EyeCantCU i think some your PRs don't match the atgreed upon times
Let me look again
I thought they did
https://github.com/ublue-os/main/issues/363#issuecomment-1788338503
i don't mind changing, i just want to be on same page
https://hackmd.io/oCQ2a6ovSe-JMXALWA5lKA
Looks like they changed? Maybe my page on hackmd didn't refresh?
how did we pick 1540?
I'm unsure
i think you were trying to stagger framework from asus/surface
well, really nvidia/framework are at asme time
I'll update the commits to match those times
Though I swear those don't match the page I have up lol
we can revise the times if they arne't good. 🙂 i just want to have a record of the plan
well i can't even get hackmd to work anymore, so maybe my copy was bad
i dunno, i'm hating hackmd right now
yeah, well, we'll watch it live tomorrow, no big deal
Yeah. I like the linked one but 35 minutes between *-nvidia and bazzite/bluefin might be a bit much
there's an entire week of waiting until F39 so plenty of time
Awesome, sounds good to me
here's what i see now
then tomorrow we can do beyond, and then longtail the unofficial stuff
because i tweaked based on comments here
though we should put budgie on the chopping block asap
yes, changes later are fine, it's just always annoying, so i'd rather get it mostly right now while we are all in the headspace ;D
@EyeCantCU does this seem good?
Yeah, this looks good. Don't mind changing it up real quick
i only cleaned up formatting of nvidia, framework
and moved bazzite/bluefin to 1640, saving 5 minutes based on your comment about 35 minutes
I'm updating the version on the issue comment too
Solid!
hey, if config/akmods/main/nvidia are good, it's all good in my book
🤣
@j0rge this one needs your rubber stamp 😉 https://github.com/ublue-os/main/pull/407
RJ and I both worked on it so I don't want to approve
@bsherman all times match the requested ones now 🙂