Why do Manjaro Stable updates break the system? Is it bad testing?

Just got the update notice and tried updating with pamac & got:

Synchronizing package databases...

Starting full system upgrade...
Preparing...
Resolving dependencies...
Checking inter-conflicts...
Downloading...
Downloading manjaro-system-20170406-4-any.pkg.tar.xz...
Error: failed retrieving file 'manjaro-system-20170406-4-any.pkg.tar.xz' from manjaro.cybr.ca : The requested URL returned error: 404 Not Found
Error: failed retrieving file 'manjaro-system-20170406-4-any.pkg.tar.xz' from ca.mirror.babylon.network : The requested URL returned error: 404 Not Found
Warning: failed to retrieve some files

Failed to commit transaction:
error invoking external downloader

Maybe three branches is much, but it could be a pita to develop for manjaro with just stable branch.

4 Likes

I been in Linux number of years. Have a lot patience for most things, However if Arch Linux ever wants to be considered a server distro this kind of thing can't happen. The one problem with Arch Linux itself is it one tier system. What I mean by that is there is no unstable distro, no testing distro, for Arch test things in before going to its stable branch. Basically it Tree for not breaking your stable branch so you have something that just works. Debian has this kind of system down pat to where you never worry updating stable at all.

Okay, sarcasm detector going off here, waiving a towel in front of it, bloody thing it is still beeping, batteries don't need to be changed, but I'll bite anyway.

The problem is currently how Testing is used.

Unstable contantly has new packages dribbling into it when they become available. This includes upstream Arch Stable packages and Manjaro managed packages like kernels, drivers and tools. All good this is what unstable is for.

Testing is currently treated almost like a secondary unstable repo, minus the development. Packages are promoted to Testing in small batches, quite often. Sometimes there is an announcement thread if something significant is promoted (ie DE update, python update, etc), but most times there is not. It is up to the testing user to keep on top of these changes and test accordingly.

When it comes time to promote these packages to stable, many of these small testing batch updates are combined into a single large batch update. This large batch update process has not been tested, and this has been the source of ALL the major Manjaro update issues since I've been using it.

@c00ter has also pointed this out in this thread, multiple times.

@anon35400795 has been banging on about this for a while, and I agree with him now, there needs to be a 1:1 mapping between batch updates applied to Testing and batch updates applied to Stable.

A more robust change management procedure is required where the exact same large batch updates that are applied in Stable get applied in Testing first.

This would have helped enormously during this libglvnd update, and another very problematic update it would have helped was the notorious systemd upgrade nightmare. In that instance there were no issues in Testing because systemd was incrementally updated from 2.31-1 to 2.31-4, but when this change was applied in a single update in Stable it crashed systems during the update and left many systems in an unrecoverable state.

Had this batch update been applied in Testing first, the issue would have been caught earlier and remedied before migration to Stable. Same with this update and the libglvnd GLX issues.

Problem is I don't know how much of an appetite there is for this type of change from within the Manjaro Team, given it would involve a little more change mangement overhead and a slightly more restrictive use of the Testing environment.

8 Likes

I see some merit to this idea. I think some posts in this thread also indicate that we could use more developers for

  1. mhwd
  2. manjaro-system or the pacman hooks to replace it.

This could help the stability of the stable repos.

3 Likes

Hi,

I did not analyse what exact change @philm did to manjaro-system but now the update works flawlessly on one of my systems. I had downgraded my system via LVM snapshot before I started the update again.

@philm YOU ARE GREAT! THANK YOU

4 Likes

I have had a look at the forum software and the possibilities it has.

Did you know that for every category including subcategory an rss feed exist?
Announcements is on

https://archived.forum.manjaro.org/c/announcements.rss

Stable-updates is on

https://archived.forum.manjaro.org/c/announcements/stable-updates.rss

Say the team creates a category which is only used for short but very important messages regarding updates.

Said category could be reached at a similar location

https://archived.forum.manjaro.org/c/announcements/very-important.rss

The feed from said category could among other things be placed a prominent place on manjaro.org.

Said feed could also be read by the pamac update-checker and be displayed a promiment place on the users computer.

7 Likes

I tend to gree that getting rid of testing wouldn't hurt.

But unstable is fine, if thats named testing after testing got removed, I don't care.

We have had lately a little discussion internally on repos and structure.

In my view, what works fine in boxit with arch packages, doesn't work so fine with manjaro exclusive packages.
This is due to the fact that boxit can't selectively make snapshots of given packages, rather to diff the entire branch.

A solution would be to add new repos within the branches for manjaro packages, no matter how many branches we have.
It means, we would separate arch packages and manjaro packages, and keep them in dedicated repos.

Point here, arch is upstream with arch packages, manjaro is upstream for manjaro packages, and both together doesn't work out so good with increasing manjaro packages.

2 Likes

If the testing repo was being used effectively it would be very useful, unlike how it is being used currently.

4 Likes

I was thinking about testing repository. it's just my 2 cents.
testing repository should be like as the name tell a repository to test "stable updates".
and it would need user that can and agree to back up or make snapshot of their system.
as devs will have to make adjustment to the repositories to correct bugs or workaround for the complicate updates like as phil did yesterday with mhwd.

I see that user on testing should "restore" the state of testing repository the last time it was all pushed to stable. and then test the future stable update from the beginning when devs push corrections in testing. and once all seems to work. push all testing to stable, and user create a new snapshot.
this to test all the process stable user will have, not piece after piece, I hope I'm clear. :wink:

don't know at all if it the right direction. :stuck_out_tongue:

2 Likes

I can only talk about my experience with deploying openrc related packages, which have become quite a lot.

I basically don't need testing at all, I use unstable ideally as long as necessary, and I deem it stable to be moved in other branches. Ironically, this makes unstable sometime more stable than stable. I only move seldom packages also to testing, but in truth, testing is just there, I don't need it, its sometime even more a burden than help.

2 Likes

I'm also on unstable and never had problems too. or any that really broke my system few things that stop working for a time.
like as I see it now..
testing repos is just a repository kind of like as unstable (but maybe a little more stable with maybe more user to detect more problems) but I think it should be more a repository to test if the stable update will be smooth or not. this would maybe need a little more work for devs, but more user implications. not sure it's really applicable in a rolling model. but..

as an analogy I could say:
Unstable should be to test if the software work (user build it and test it) and to create the packages.
and testing a repository to test if the package (installer) work and is ready for the public distribution.

3 Likes

I happen to agree with @artoo

When I switched my attention to Manjaro it was the 'Arch' OOB experience - and the possibility of having more kernels - anyone with VMware experience know why.

I use only what Manjaro calls unstable since that is what Arch considers stable.

I can live with the twitches that comes from Arch since I would have gotten them anyway and I am capable of working it out.

I also think it will give newcomers to Manjaro a clear picture of what it means to have a buffer-repo.

Also Manjaro will get more testing response than before because it will all be on one repo testing or unstable what ever you want to call it.

2 Likes

That would be in practice a nightmare with current boxit structure.

I would prefer additional repos.
If you imagine snapshots to be vertically, branch to branch, moving from repos(like arch does) could work out fine for manjaro specific packages only. We would move them repo to repo horizontally, within the branch.

Example:

I would move packages from a hypothetical [openrc-staging] repo within unstable to [openrc] repo.

But Manjaro does, and their users certainly do.

All the major update issues seem to be caused by large batch stable updates that are not tested beforehand. From a change management and quality control perspective it just doesn't make any sense.

I agree, keep unstable the way it is, for development and adding upstream packages. But when it comes time for a stable release, do a dry run in testing first. Make this function testing's primary purpose. Make sure there are no hidden demons, no serious hidden side effects, and no serious bugs in the batch update.

When the all clear is declared in testing then apply exactly the same update into stable.

At the very least this should greatly reduce the possibility of Manjaro updates causing potentially serious issues to their users' systems.

5 Likes

Frankly, you people have no idea what you ask for. We simply do not have the man power and resources to test this all, its why I think cluttering in 3 branches falls us back on our feet.

3 branches means triple QA, not good, we are unable to achieve such thing. Dang, there I said it, hard truth.

1 Like

Incorrect, I know exactly what I ask for.

Exactly what is occurring now, only instead of batch updates trickling into testing willy nilly with no structure or purpose, we add some structure and purpose. Very little would change, just there would be only be an update to testing prior to an intended stable release.

You guys release an update that borks many users systems and your response is to propose reducing testing of your releases?

That is just nuts and borderline irresponsible.

6 Likes

Do you volunteer for anything of that? Be responsible, come on, contribute.
Or, if you take care my fridge isn't empty, I could allocate much more time for manjaro, instead of doing other stuff to fill my fridge.
Guys, we do all this in our freaking free and spare time.

6 Likes

I think this is a good idea. This would mean pushing a Testing update every 2-3 weeks several days before a Stable update, then testers can test if it breaks the system. This would not be a problem for me at all, because I would just create a btrfs snapshot and update the batch, then report. (Others would use Clonezilla, VM, LVM, ZFS, Nilfs2 or just rely on their experience to fix anything.)

The way it is now testers rather test individual packages and they do not test the systemic stability of a batch update.

But @philm has to consider if it is viable to get such a "Stable RC" testing.

7 Likes

What I am proposing is almost identical to the current setup, only less testing releases.

Very little will change, only how the testing releases are put together. Less often and with the purpose of actually testing a stable release.

Users will be asked to do most of this testing, via announcement thread, like they currently are.

I imagine @philm will put together the releases just like he currently does.

I already have systems to test unstable and testing updates. I check them every day.

I have been tirelessly testing manjaro-architect, burning the midnight oil for weeks on end.

I attempt to help users on this forum that have issues. Pretty much every day.

I am not a developer but I do contribute where I can.

I will be involved in testing these releases, if they actually happen.

And yes I have a full time job too, and a family too.

12 Likes

Forum kindly sponsored by