My local repository under Manjaro

AlanP · 17 March 2020 14:04

Hello,

I come back to you to get help to add my local repository under Manjaro. I followed instructions given by some users here but I meet some difficulties.

First, I got all the packages and directories via the rsync tool.

Here is how my local repository is defined on my external hard drive after recovering via rsync:

/nameofmyexternalHD/Manjaro/localrepository/MJRO1902/

/pool
->/overlay
  ->Packages
->/overlay-32
  ->Packages
->/sync
  ->Packages
->/sync-32
  -->Packages

/stable
->/community
  ->/i686
    ->community.db
    ->community.db.tar.gz
    ->community.files
    ->community.files.tar.gz
  ->/x86_64
    ->Packages
->/core
  ->/i686
    ->core.db
    ->core.db.tar.gz
    ->core.files
    ->core.files.tar.gz
    ->manjaro-system-20171199-1-any.pkg.tar.xz
    ->manjaro-system-20171199-1-any.pkg.tar.xz.sig
  ->/x86_64
    ->Packages
->/extra
  ->/i686
    ->extra.db
    ->extra.db.tar.gz
    ->extra.files
    ->extra.files.tar.gz
  ->/x86_64
    ->Packages
->/multilib
  ->/x86_64
    ->Packages
state

/stable-staging
->/community
  ->/i686
    ->community.db
    ->community.db.tar.gz
    ->community.files
    ->community.files.tar.gz
  ->/x86_64
    ->Packages
->/core
  ->/i686
    ->core.db
    ->core.db.tar.gz
    ->core.files
    ->core.files.tar.gz
    ->manjaro-system-20171199-1-any.pkg.tar.xz
    ->manjaro-system-20171199-1-any.pkg.tar.xz.sig
  ->/x86_64
     ->Packages
->/extra
  ->/i686
    ->extra.db
    ->extra.db.tar.gz
    ->extra.files
    ->extra.files.tar.gz
  ->/x86_64
    ->Packages
->/multilib
  ->/x86_64
    ->Packages
state

/x32-stable
->/community
  ->/i686
    ->Packages
->/core
  ->/i686
    ->Packages
->/extra
  ->/i686
    ->Packages
state

mirrorlist
state

I know I must use the repo-add command to generate the database for my local repository. But with this kind of structure I don't know how to do that ? And how to add it in the pacman.conf file ? That's why I posted this message.

Have you got an idea ?

Thanks a lot for your help.

Regards.

Fabby · 17 March 2020 18:01

Looking at this and your previous question, this looks like an XY problem... Could you please explain in plain English what you're trying to accomplish without using the words "mirror" and "sync"?

We thank you for your cooperation...

AlanP · 18 March 2020 13:09

Hello,

it is simple. I try to achieve a local repository on my external hard disk drive to make updates and install programs on computer who don't have internet connection. I have already done this on Linux distributions like Linux Mint and Fedora without too much problems but on Manjaro it is the first time. To make my local repository I must download Manjaro packages, and the rsync command help me to do that. This allows to recover the content of an existing and remote mirror for Manjaro on an external support (like usb key or external hard drive for example). Following packages recovery, I get a local 'structured' repository (see my message above). As you can see in the first link you given, some people told me how to recover the packages and how to add my local repository in Manjaro (with the help of the repo-add command). But here I don't have one folder but many. In fact, I've got a 'copy' of the remote mirror where I downloaded the packages on my external hard drive (for the stable branch). Now what I want to know is how to define that 'copy' as a local repository to be able to updates and install programs on my Manjaro system without internet (by using my external hard disk) ?

I hope that my explanation is understandable.

Regards.

Note: I made some research on this forum, and I think I found the solution that I'm looking for (see it). Thanks to xabbu.
In my case, the repo-add command seems to be not necessary. I only have to define the path for Core, community, extra, multilib with my external hard disk. I will test it as soon as possible and I will tell you if it worked. I hope so.

Fabby · 18 March 2020 17:42

It makes sense if you want to have a local repository for multiple computers, so:

dump the mirror on a local NAS,
have it available on an IP address via HTTP(s) or FTP

add your NAS to the following file:

/var/lib/pacman-mirrors/custom-mirrors.json

and don't forget to rsync regularly!

AlanP · 19 March 2020 13:15

Hi,

thanks for your reply.

I don't have a NAS, and I don't want to use the network to update my machine. I want to do it locally via an external hard drive. Well, I tested the solution given in the link of my previous message (xabbu solution). It works but not as I would like. Let me explain. Via the terminal, there is no problem, I can update the system, and install programs with the appropriate pacman commands. But what I want is to update the system, and install the programs via the graphical interface (pacmac). And there, it gets stuck.

I get many error messages with this. As:

"failed to synchronize all databases"
"failed to retrieve some files"
"file: ///run/media/..../nameofmyharddisk/localrepository/MJRO1902/stable/core/x86_64/core.db: Error resolving “ ”: Name or service not known"

I really want to use my local repository with pacmac. The software installed by default to add/remove/ update software on Manjaro (in graphical mode).

Is there a way to make my local repository available on my external hard drive work with pacmac?

Thank you.

Fabby · 19 March 2020 13:22

I'm coming back to my previous statement:

Running a full mirror for a single machine doesn't make sense because updating the mirror will take Terabytes of transmission whereas the updates themselves will only take Gigabytes so you'll have to run through hoops by creating an HTTP or FTP server on your own machine and then have it update from itself over its internal network as any mirror is always run over the network, not from a local disk.

What are you trying to accomplish here? Forget about the solution you think you have.
Please state your problem.

AlanP · 19 March 2020 13:30

I want to do this because I want to install Manjaro on the personal computer of a member of my family who does not have internet. So the only solution is to use an external support (like hard disk). I have already done this kind of thing on distributions like Linux Mint, Ubuntu or Fedora without too much problem. But under Manjaro, it's a little more complicated.

Fabby · 19 March 2020 13:42

The only solutions I can think of are:

Solution 1:

Install the computer at your place
Bring the computer to the family member

Solution 2:

Buy a NAS without HDD
install your HDD into it
Bring NAS to Family member
Install there

Solution 3:

Turn your computer into an FTP server
Take your computer to family member
Install there

Not really a solution 4:

You could try to add a local disk URL to the mirror list without a protocol in the /var/lib/pacman-mirrors/custom-mirrors.json but you're on your own there as I've never needed something like that nor will ever need it so you're on your own there!
Good luck! (You'll need it!)

¯\_(ツ)_/¯

linux-aarhus · 19 March 2020 13:47

If the system works as expected then updates are not required.

AlanP · 19 March 2020 13:51

I will take the fourth solution.

I see it on the forum link. I think it is the best solution for me but the command line is not displayed in the first message.

If you have tips for creating a custom mirror, I'll take it.

AlanP · 19 March 2020 13:53

Ok, But how can i install software without internet ?

linux-aarhus · 19 March 2020 14:11

[HowTo] Create a Manjaro mirror server using Architect

Locate a mirror offering rsync - you may already have done that.

Then rsync the branch the you want to use into a folder of the same name.

Sample script

#!/bin/bash
HOME="/volume1/web"
TARGET="${HOME}/public/manjaro"
TMP="${HOME}/.tmp/manjaro"
LOCK="/tmp/rsync-manjaro-all.lock"
SOURCE="rsync://mirrorservice.org/repo.manjaro.org/repos"

[ ! -d "${TARGET}" ] && mkdir -p "${TARGET}"
[ ! -d "${TMP}" ] && mkdir -p "${TMP}"

exec 9>"${LOCK}"
flock -n 9 || exit

if ! stty &>/dev/null; then
    QUIET="-q"
fi

rsync -rtlvH --safe-links \
    --bwlimit=${BWLIMIT} \
    --delete-after --progress \
    -h ${QUIET} --timeout=600 --contimeout=120 -p \
    --delay-updates --no-motd \
    --temp-dir="${TMP}" \
    --exclude="/unstable" \
    --exclude="/testing" \
    --exclude="stable-staging" \
    ${SOURCE} \
    "${TARGET}"

Copy the folder to an USB device.

On the target system mount the device.

Modify the target systems mirror list to point to your file based repo.

AlanP · 19 March 2020 14:12

I have a 'copy' of one Manjaro mirror on my external hard disk.

Modify the target systems mirror list to point to your file based repo.

So I have to define my local repository as a mirror if I want to use it (with pacmac for example), if I understood correctly. Ok. But now how can I do that ? I don't know anything about this kind of thing.

Thanks.

linux-aarhus · 24 March 2020 14:20

IF the copy you have is a mirror of a branch - which includes subfolders and the databases from the mirror

stable
- community
  - x86_64
- core
  - x86_64
- multilib
  - x86_64
- extra
  - x86_64

Edit /etc/pacman.d/mirrorlist

Copy the first line, modify and save the file

Server = file:///path/to/mountpoint/stable/$repo/$arch

If you have done it correct - you can just do

sudo pacman -Syu some-package

AlanP · 19 March 2020 14:21

IF the copy you have is a mirror of a branch - which includes subfolders

branch
    community
        x86_64
    core
        x86_64
    multilib
        x86_64
    extra
        x86_64

Edit /etc/pacman.d/mirrorlist

Copy the first line and modify the line and save the file

Server = file:///path/to/mountpoint/stable/$repo/$arch

If you have done it correct - you can just do

sudo pacman -Syu some-package

I've done it. It works under the terminal (with the pacman commands) but not with pacmac software (see my third message above).

linux-aarhus · 19 March 2020 14:29

There is no other ways.

If you change the mirror list as I mention - to use a file based mirror - pamac doesn't care - pamac uses libalpm which uses the mirror list.

You could just remove the additional content leaving only the file based mirror.

Otherwise report it as bug on https://gitlab.manjaro.org/applications/pamac/

AlanP · 19 March 2020 14:32

Thanks for your help.

I will try it again tomorrow. I didn't think it would be so complicated under Manjaro.

Otherwise, I will use the tips for custom-mirrors.

Regards.

linux-aarhus · 19 March 2020 15:05

It is not complicated - you just have to know - just like everything else.

The simple uncomplicated solution is to edit your mirrorlist

Server = file:///path/to/mirror/stable/$repo/$arch

Everything below this is complications.

Don't - pacman-mirrors does not recognize the file:// protocol - so setting a mirror using the custom-mirrors.json will not work.

If you want to avoid editing your mirror list by hand then use the api

sudo pacman-mirrors --api --url file:///path/to/mirror

Still mask the pamac services/timers because pamac may do things on its own.

systemctl list-unit-files | grep pamac

sudo systemctl mask pamac-cleancache.service 
sudo systemctl mask pamac-mirrorlist.service
sudo systemctl mask pamac-mirrorlist.timer
sudo systemctl mask pamac-cleancache.timer

handy · 20 March 2020 00:17

@AlanP, you could clone a Manjaro install to a drive (or an image - much smaller) & then install that on the machine that has no internet.

I've done it multiple times over the ~15 years. It has only become easier as the years go by. Clonezilla is my friend.

Beyond that MX & antiX Linux have fantastic USB live tools, where you can simply create custom installs that have permanent storage on USB stick or HDD.

AlanP · 20 March 2020 12:42

Hi,

I followed your instructions. My local repository is well define in the /etc/pacman.d/mirrorlist file (it is the only one in the list). I launched the sudo pacman -Syyu command after. I also masked the pamac services/timers as you told to me. So all things are done. Despite it, I can't install or update programs under pamac (I tried to install libreoffice-fresh with the extensions (math,...)). Each time I do it, I've got the following error popup messages :

"failed to synchronize all databases"

"failed to retrieve some files"

"file: ///run/media/..../nameofmyharddisk/localrepository/MJRO1902/stable/core/x86_64/core.db: Error resolving “ ”: Name or service not known"
"file: ///run/media/..../nameofmyharddisk/localrepository/MJRO1902/stable/community/x86_64/community.db: Error resolving “ ”: Name or service not known"
"file: ///run/media/..../nameofmyharddisk/localrepository/MJRO1902/stable/extra/x86_64/extra.db: Error resolving “ ”: Name or service not known"
"file: ///run/media/..../nameofmyharddisk/localrepository/MJRO1902/stable/multilib/x86_64/multilib.db: Error resolving “ ”: Name or service not known"

I also want to tell you that some dependencies or files are available in the staging-stable branch (see my first message). It is the case for libreoffice who need some files/dependencies from that branch.

If you have an idea on what's wrong. Thanks for your help.

Regards.