Pacman-Mirrors 4.1.0-dev

Announcements
,
#1

April . 2017 v4.1.0-dev

v4.1.0-dev

  • Simple API: packagers are the intended target @artoo
    • add a prefix to be used for files for example $MNT/etc/pacman-mirrors.conf
    • read/write branch in pacman-mirrors
    • write protocols in pacman-mirrors.conf
  • Protocols option in pacman-mirrors.conf
    • Control which protocols pacman-mirrors can use
    • Protocols are used in the order in which they appear
      • available: http,https,ftp
      • example: Protocols = http,https
      • example: Protocols = https,http
      • unmentioned protocols are not used nor ranked
  • General improvement if a mirror offers several protocols
    • protocols are reverse sorted
    • ssl enabled mirrors will be prioritized
  • minor bugfixes
~ >>> pacman-mirrors -h                                                        
usage: pacman-mirrors [-h] [-g] [-m {rank,random}]
                      [-b {stable,testing,unstable}] [-c COUNTRY [COUNTRY ...]
                      | --geoip] [-d PATH] [-o FILE] [-t SECONDS]
                      [--no-update] [-i] [-v] [-q] [-f DIGIT] [-l] [--default]
                      [-a] [--get-branch | --set-branch] [--prefix PREFIX]
                      [--proto PROTO [PROTO ...]]

optional arguments:
  -g, --generate        Generate mirrorlist
  -m, --method {rank,random}
                        Generation method
  -b, --branch {stable,testing,unstable}
                        Branch name
  -c, --country COUNTRY [COUNTRY ...]
                        Comma separated list of countries, from which mirrors
                        will be used
  --geoip               Get current country using geolocation. Ignored if
                        '-c/--country' is supplied
  -d, --mirror_dir PATH
                        Mirrors list path
  -o, --output FILE     Output file
  -t, --timeout SECONDS
                        Maximum waiting time for server response
  --no-update           Don't generate mirrorlist if 'NoUpdate = True' in the
                        configuration file
  -i, --interactive     Generate custom mirrorlist
  -v, --version         Print the pacman-mirrors version
  -q, --quiet           Quiet mode - less verbose output
  -f, --fasttrack DIGIT
                        A quick mirrorlist. Overrides -c -i -m --geoip
  -l, --list            List all available countries
  --default             Load default mirror file

API:
  -a, --api             [--prefix] [--protocols] [{-b }--set-
                        branch|--get-branch]
  --get-branch          get-branch
  --set-branch          set-branch
  --prefix PREFIX       {'$MNT' | '/some/path'}
  --proto PROTO [PROTO ...]
                        {none | http https ftp ftps}
16 Likes
Synchronizing package databases error - servers down?
How to create a Custom mirrorlist - the NEW .json way
Manjaro-architect 0.8
Cannot Switch to Testing
[Stable Update] 2017-04-02 - Mesa-Stack, Kernels, Plasma, Firefox
How to create a Custom mirrorlist - the NEW .json way
[Testing Update] 2017-03-22 - Elogind, Plasma, Deepin, Manjaro-Architect
Installation with Manjaro-Architect ISO
Mirrors Jam
[Stable Update] 2017-04-02 - Mesa-Stack, Kernels, Plasma, Firefox
[Résolu]Erreur synchronisation
#2

I expect you to try and break it - use weird combination of arguments.

Please report back if you encounter errors :slight_smile:

#3

I prefer to do -Syy instead of -Syu .. to prevent the full upgrade for newbie :wink:

2 Likes
#4

Cool stuff!! Thanks a lot! will test immediately... :sunglasses:

#5

I have a simpler suggestion still. You can just install a single package directly from a mirror (as long as it's a package where you don't have to be concerned about dependencies/compatabilities, mind you!)

Goes like this in case of pacman-mirrorlist-dev:

sudo pacman -U http://manjarolinux.polymorf.fr/pool/overlay/pacman-mirrorlist-dev-20170311-2-any.pkg.tar.xz
1 Like
#6

Thanks for testing - please try and break it :slight_smile:
You're right but that implies that you know the full packagename :slight_smile:

#7

Here's my first result, interesting! :wink:

$ sudo pacman-mirrors -f5
.: Info ping google.com x 3
.: Info ping repo.manjaro.org x 1
.: Info Downloading mirrors from repo.manjaro.org
.: Info: Querying mirrors - This may take some time
   0.224 Denmark        : 00:08: http://www.uex.dk/repos/manjaro/
   ..... Germany        : 00:09: https://mirror.netzspielplatz.de/manjaro/   0.604 
   0.468 Germany        : 00:09: https://repo.stdout.net/manjaro/
   0.500 Germany        : 00:09: https://manjaro-mirror-a.alpix.eu/
   ..... Japan          : 00:09: http://ftp.tsukuba.wide.ad.jp/Linux/manja   0.767 
.: Info Writing mirror list
   Denmark         : http://www.uex.dk/repos/manjaro/stable/$repo/$arch
   Germany         : http://repo.stdout.net/manjaro/stable/$repo/$arch
   Germany         : https://repo.stdout.net/manjaro/stable/$repo/$arch
   Germany         : http://manjaro-mirror-a.alpix.eu/stable/$repo/$arch
   Germany         : https://manjaro-mirror-a.alpix.eu/stable/$repo/$arch
   Germany         : https://mirror.netzspielplatz.de/manjaro/packages/stable/$repo/$arch
   Japan           : ftp://ftp.tsukuba.wide.ad.jp/Linux/manjaro/stable/$repo/$arch
   Japan           : http://ftp.tsukuba.wide.ad.jp/Linux/manjaro/stable/$repo/$arch
.: Info Mirrorlist generated and saved to: /etc/pacman.d/mirrorlist

$ cat /etc/pacman.d/mirrorlist
##
## Manjaro Linux mirrorlist
## Generated on 11 March 2017 13:55
##
## Use pacman-mirrors to modify
##

## Country       : Denmark
Server = http://www.uex.dk/repos/manjaro/stable/$repo/$arch

## Country       : Germany
Server = http://repo.stdout.net/manjaro/stable/$repo/$arch

## Country       : Germany
Server = https://repo.stdout.net/manjaro/stable/$repo/$arch

## Country       : Germany
Server = http://manjaro-mirror-a.alpix.eu/stable/$repo/$arch

## Country       : Germany
Server = https://manjaro-mirror-a.alpix.eu/stable/$repo/$arch

## Country       : Germany
Server = https://mirror.netzspielplatz.de/manjaro/packages/stable/$repo/$arch

## Country       : Japan
Server = ftp://ftp.tsukuba.wide.ad.jp/Linux/manjaro/stable/$repo/$arch

## Country       : Japan
Server = http://ftp.tsukuba.wide.ad.jp/Linux/manjaro/stable/$repo/$arch

Danmark and Japan included, wow! :smiley:
I live in Austria ... but doesn't mean much of course...

1 Like
#8

Ranking works surprisingly fast, impressive! :thumbsup:

1 Like
#9

you can look it up on your favourite mirror :wink:
But I mean more for announcements like here you can use it - as an internal more or less...

1 Like
#10

I'm doing my best! :wink: :laughing:

.: Info ping google.com x 3
.: Info ping repo.manjaro.org x 1
.: Info Downloading mirrors from repo.manjaro.org
.: Info: Querying mirrors - This may take some time
   0.010 Denmark        : 00:08: http://www.uex.dk/repos/manjaro/

.: Error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) 'https://mirror.netzspielplatz.de/manjaro/packages/'
   ..... Germany        : 00:09: https://mirror.netzspielplatz.de/manjaro/packages/
.: Error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) 'https://repo.stdout.net/manjaro/'
   ..... Germany        : 00:09: https://repo.stdout.net/manjaro/
.: Error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) 'https://manjaro-mirror-a.alpix.eu/'
   0.019 Germany        : 00:09: https://manjaro-mirror-a.alpix.eu/   ..... Japan          : 00:09: http://ftp.tsukuba.wide.ad.jp/Linux/manjaro/
   0.015 Bulgaria       : 00:18: http://manjaro.ipacct.com/manjaro/
   0.016 France         : 00:29: http://manjarolinux.polymorf.fr/
   0.015 France         : 00:29: http://mirror.lignux.com/manjaro/
.: Info Writing mirror list
   Denmark         : http://www.uex.dk/repos/manjaro/stable/$repo/$arch
   Bulgaria        : http://manjaro.ipacct.com/manjaro/stable/$repo/$arch
   France          : http://mirror.lignux.com/manjaro/stable/$repo/$arch
   France          : http://manjarolinux.polymorf.fr/stable/$repo/$arch
   Japan           : ftp://ftp.tsukuba.wide.ad.jp/Linux/manjaro/stable/$repo/$arch
   Japan           : http://ftp.tsukuba.wide.ad.jp/Linux/manjaro/stable/$repo/$arch
.: Info Mirrorlist generated and saved to: /etc/pacman.d/mirrorlist

This happens when my mobile wifi is not logged into a network or has very bad reception while the computer is of course still connected fine to the wifi. For some reason pinging google doesn't fail even though in fact it cannot succeed .... or at least I wouldn't know how Google still manages to do that - well you never know! :wink: ... :open_mouth:

The resulting mirrorlist in any case is fun!??

#11

I am thinking of a way to combine --country or --geoip without sacrificing too much in regard to execution time.

The --fasttrack argument was initially thought as a help for installers to quickly get

  1. up-to-date mirrorlist
  2. as responsive as possible

So if a mirror for some reason is inaccessible as in your result the requested number of mirrors from the argument is guaranteed

2 Likes
#12

Clever! So can you explain how it chose the Bulgarian and French mirrors in my case - just out of interest...
it ceratainly didn't have time to go through any complete list for that.

#13

This made me dizzy.
As I am understanding it:
Your computer is tethering on your mobiles wifi?
And when your mobiles network connection is poor or even down - it is still possible to ping google?

The piece of code that does the pinging is using your systems ping utility so your phone is with android?

def ping_host(host, count=1):
    """Check a hosts availability
    :param host:
    :param count:
    :rtype: boolean
    """
    print(".: {} ping {} x {}".format(txt.INF_CLR, host, count))
    return system_call("ping -c{} {} > /dev/null".format(count, host)) == 0
#14

sorry, you explicitly asked for it! :laughing:

No phone. it's just a mobile wifi, similar to something like this.

As far as I know, no android, really, although it seems to use some kind of a basic system and is able to produce an html login page ...
For the computer it's just like any other wifi.
In the example posted above the hotspot was logged out of the network, like you would plug out the network cable of your router with the wifi still running.
Pinging Google should definitely not be possilble - I honestly hope!

#15

When you look at the status.json from repo.manjaro.org you will find that the dictionary for each mirror contains more info than the mirrors.json

There are one element which the fasttrack argument use

"branches": x, x, x

this element have different value depending on how up-to-date the mirror is so pacman-mirrors

  1. creates a filtered list with "branches": 1, 1, 1.
  2. randomizes the up2date mirrors (evens the distribution on all up2date mirrors)
  3. ranks the list by checking response time
  • if a mirror responds within the default timeout of 2s
  • a counter gets +1 and the mirror is copied to worklist
  • when the count reaches -f the worklist is written as a mirrorlist
1 Like
#16

It's a very cool and useful addition. I like it a lot. Thank you very much for this contribution!
Don't grow any grey hair about the mobilewifi stuff, please! :wink: It's an exotic scenario and it has nothing to do with the features of pacman-mirrors -f

1 Like
#17

If I did it wouldn't be visible as it is the color

#18

Same here! :laughing:
Also about the pinging I had already run into the same strange issue when I built almost exactly that same connection switch for manjaro-architect just recently.
I just gave up and decided it was an oddity of my hardware... :wink:

#19

Why by default you track only stable ? You not read the setting into the pacman-mirrors.conf to read what branch the user use ?

#20

The users default branch is used no matter where it is set - exactly as if you ran with -g

Forum kindly sponsored by