BeautifulSoup runs horribly slower on Manjaro than on Windows

Python's BeautifulSoup package runs significantly faster( Seconds vs. Minutes) on my old Windows laptop(Intel i3 4GB RAM) than on my new Linux computer (Intel i7 32GB RAM).

How could that be?

I've encountered a discussion about it- but it didn't solve my problem:

Sincerely yours,
Me :slight_smile:

Welcome.


Start with

inxi -Fxxxrza --no-host

as always @c00ter said :smiley: :slight_smile:

System:
  Kernel: 5.4.12-1-MANJARO x86_64 bits: 64 compiler: gcc v: 9.2.0 
  parameters: BOOT_IMAGE=/boot/vmlinuz-5.4-x86_64 
  root=UUID=7db98f53-a54f-4c21-b2e4-9fff71a772c8 rw quiet apparmor=1 
  security=apparmor resume=UUID=a3d0a02e-e68c-46d1-a9cd-3d271526bf34 
  udev.log_priority=3 
  Desktop: Xfce 4.14.2 tk: Gtk 3.24.13 info: xfce4-panel wm: xfwm4 
  dm: LightDM 1.30.0 Distro: Manjaro Linux 
Machine:
  Type: Laptop System: LENOVO product: 81UH v: Lenovo Legion Y740-15IRHg 
  serial: <filter> Chassis: type: 10 v: Lenovo Legion Y740-15IRHg 
  serial: <filter> 
  Mobo: LENOVO model: LNVNB161216 v: No DPK serial: <filter> UEFI: LENOVO 
  v: BVCN10WW(V1.06) date: 06/06/2019 
Battery:
  ID-1: BAT1 charge: 18.3 Wh condition: 54.7/57.0 Wh (96%) volts: 10.5/11.5 
  model: 0000313100504D53 324750334D37314C type: Li-ion serial: <filter> 
  status: Discharging 
CPU:
  Topology: 6-Core model: Intel Core i7-9750H bits: 64 type: MT MCP 
  arch: Kaby Lake family: 6 model-id: 9E (158) stepping: A (10) 
  microcode: CA L2 cache: 12.0 MiB 
  flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx 
  bogomips: 62431 
  Speed: 1000 MHz min/max: 800/4500 MHz Core speeds (MHz): 1: 1000 2: 1003 
  3: 1000 4: 1000 5: 1000 6: 1000 7: 1000 8: 1000 9: 1001 10: 1000 11: 1001 
  12: 1000 
  Vulnerabilities: Type: itlb_multihit status: KVM: Split huge pages 
  Type: l1tf 
  mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable 
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable 
  Type: meltdown mitigation: PTI 
  Type: spec_store_bypass 
  mitigation: Speculative Store Bypass disabled via prctl and seccomp 
  Type: spectre_v1 
  mitigation: usercopy/swapgs barriers and __user pointer sanitization 
  Type: spectre_v2 mitigation: Full generic retpoline, IBPB: conditional, 
  IBRS_FW, STIBP: conditional, RSB filling 
  Type: tsx_async_abort status: Not affected 
Graphics:
  Device-1: NVIDIA TU106BM [GeForce RTX 2060 Mobile] vendor: Lenovo 
  driver: nvidia v: 418.113 bus ID: 01:00.0 chip ID: 10de:1f51 
  Display: x11 server: X.Org 1.20.7 driver: nvidia 
  resolution: 1920x1080~144Hz 
  OpenGL: renderer: GeForce RTX 2060/PCIe/SSE2 v: 4.6.0 NVIDIA 418.113 
  direct render: Yes 
Audio:
  Device-1: Intel Cannon Lake PCH cAVS vendor: Lenovo driver: snd_hda_intel 
  v: kernel bus ID: 00:1f.3 chip ID: 8086:a348 
  Device-2: NVIDIA TU106 High Definition Audio vendor: Lenovo 
  driver: snd_hda_intel v: kernel bus ID: 01:00.1 chip ID: 10de:10f9 
  Sound Server: ALSA v: k5.4.12-1-MANJARO 
Network:
  Device-1: Intel Wireless-AC 9560 [Jefferson Peak] vendor: Bigfoot Networks 
  driver: iwlwifi v: kernel bus ID: 00:14.3 chip ID: 8086:a370 
  IF: wlp0s20f3 state: up mac: <filter> 
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet 
  vendor: Lenovo driver: r8169 v: kernel port: 3000 bus ID: 3f:00.0 
  chip ID: 10ec:8168 
  IF: enp63s0 state: down mac: <filter> 
Drives:
  Local Storage: total: 1.84 TiB used: 7.40 GiB (0.4%) 
  ID-1: /dev/nvme0n1 vendor: Samsung model: MZVLB1T0HALR-000L2 
  size: 953.87 GiB block size: physical: 512 B logical: 512 B 
  speed: 31.6 Gb/s lanes: 4 serial: <filter> rev: 5L1QEXA7 scheme: GPT 
  ID-2: /dev/sda vendor: Seagate model: ST1000LM049-2GH172 size: 931.51 GiB 
  block size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s 
  rotation: 7200 rpm serial: <filter> rev: LXM3 
Partition:
  ID-1: / raw size: 936.46 GiB size: 920.77 GiB (98.32%) 
  used: 7.40 GiB (0.8%) fs: ext4 dev: /dev/nvme0n1p2 
  ID-2: swap-1 size: 17.11 GiB used: 0 KiB (0.0%) fs: swap 
  swappiness: 60 (default) cache pressure: 100 (default) dev: /dev/nvme0n1p3 
Sensors:
  System Temperatures: cpu: 29.0 C mobo: N/A gpu: nvidia temp: 30 C 
  Fan Speeds (RPM): N/A 
Repos:
  Active pacman repo servers in: /etc/pacman.d/mirrorlist 
  1: http://mirror.is.co.za/mirrors/manjaro.org/stable/$repo/$arch
  2: http://manjaro.mirrors.theom.nz/stable/$repo/$arch
  3: https://www.mirrorservice.org/sites/repo.manjaro.org/repos/stable/$repo/$arch
Info:
  Processes: 285 Uptime: 2m Memory: 15.55 GiB used: 1.15 GiB (7.4%) 
  Init: systemd v: 242 Compilers: gcc: 9.2.0 Shell: bash v: 5.0.11 
  running in: xfce4-terminal inxi: 3.0.37

Try NVIDIA 430, 435 or 440

grafik

Right click deinstall 418 then right click install 4xx

You might speed up things significantly by installing the python-cchardet library from community repository.

1 Like

I was sure I did it right:
Screenshot_2020-01-22_00-15-18

But when I re-ran your code, it was as if it was undone:

System:
  Kernel: 5.4.12-1-MANJARO x86_64 bits: 64 compiler: gcc v: 9.2.0 
  parameters: BOOT_IMAGE=/boot/vmlinuz-5.4-x86_64 
  root=UUID=7db98f53-a54f-4c21-b2e4-9fff71a772c8 rw quiet apparmor=1 
  security=apparmor resume=UUID=a3d0a02e-e68c-46d1-a9cd-3d271526bf34 
  udev.log_priority=3 
  Desktop: Xfce 4.14.2 tk: Gtk 3.24.13 info: xfce4-panel wm: xfwm4 
  dm: LightDM 1.30.0 Distro: Manjaro Linux 
Machine:
  Type: Laptop System: LENOVO product: 81UH v: Lenovo Legion Y740-15IRHg 
  serial: <filter> Chassis: type: 10 v: Lenovo Legion Y740-15IRHg 
  serial: <filter> 
  Mobo: LENOVO model: LNVNB161216 v: No DPK serial: <filter> UEFI: LENOVO 
  v: BVCN10WW(V1.06) date: 06/06/2019 
Battery:
  ID-1: BAT1 charge: 10.5 Wh condition: 54.7/57.0 Wh (96%) volts: 11.9/11.5 
  model: 0000313100504D53 324750334D37314C type: Li-ion serial: <filter> 
  status: Charging 
CPU:
  Topology: 6-Core model: Intel Core i7-9750H bits: 64 type: MT MCP 
  arch: Kaby Lake family: 6 model-id: 9E (158) stepping: A (10) 
  microcode: CA L2 cache: 12.0 MiB 
  flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx 
  bogomips: 62431 
  Speed: 3546 MHz min/max: 800/4500 MHz Core speeds (MHz): 1: 4082 2: 3736 
  3: 4260 4: 3992 5: 4273 6: 4292 7: 4295 8: 3986 9: 2917 10: 3965 11: 3973 
  12: 3981 
  Vulnerabilities: Type: itlb_multihit status: KVM: Split huge pages 
  Type: l1tf 
  mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable 
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable 
  Type: meltdown mitigation: PTI 
  Type: spec_store_bypass 
  mitigation: Speculative Store Bypass disabled via prctl and seccomp 
  Type: spectre_v1 
  mitigation: usercopy/swapgs barriers and __user pointer sanitization 
  Type: spectre_v2 mitigation: Full generic retpoline, IBPB: conditional, 
  IBRS_FW, STIBP: conditional, RSB filling 
  Type: tsx_async_abort status: Not affected 
Graphics:
  Device-1: NVIDIA TU106BM [GeForce RTX 2060 Mobile] vendor: Lenovo 
  driver: nvidia v: 418.113 bus ID: 01:00.0 chip ID: 10de:1f51 
  Display: x11 server: X.Org 1.20.7 driver: nvidia 
  resolution: 1920x1080~144Hz 
  OpenGL: renderer: N/A v: N/A direct render: N/A 
Audio:
  Device-1: Intel Cannon Lake PCH cAVS vendor: Lenovo driver: snd_hda_intel 
  v: kernel bus ID: 00:1f.3 chip ID: 8086:a348 
  Device-2: NVIDIA TU106 High Definition Audio vendor: Lenovo 
  driver: snd_hda_intel v: kernel bus ID: 01:00.1 chip ID: 10de:10f9 
  Sound Server: ALSA v: k5.4.12-1-MANJARO 
Network:
  Device-1: Intel Wireless-AC 9560 [Jefferson Peak] vendor: Bigfoot Networks 
  driver: iwlwifi v: kernel bus ID: 00:14.3 chip ID: 8086:a370 
  IF: wlp0s20f3 state: up mac: <filter> 
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet 
  vendor: Lenovo driver: r8169 v: kernel port: 3000 bus ID: 3f:00.0 
  chip ID: 10ec:8168 
  IF: enp63s0 state: down mac: <filter> 
Drives:
  Local Storage: total: 1.84 TiB used: 7.66 GiB (0.4%) 
  ID-1: /dev/nvme0n1 vendor: Samsung model: MZVLB1T0HALR-000L2 
  size: 953.87 GiB block size: physical: 512 B logical: 512 B 
  speed: 31.6 Gb/s lanes: 4 serial: <filter> rev: 5L1QEXA7 scheme: GPT 
  ID-2: /dev/sda vendor: Seagate model: ST1000LM049-2GH172 size: 931.51 GiB 
  block size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s 
  rotation: 7200 rpm serial: <filter> rev: LXM3 
Partition:
  ID-1: / raw size: 936.46 GiB size: 920.77 GiB (98.32%) 
  used: 7.66 GiB (0.8%) fs: ext4 dev: /dev/nvme0n1p2 
  ID-2: swap-1 size: 17.11 GiB used: 0 KiB (0.0%) fs: swap 
  swappiness: 60 (default) cache pressure: 100 (default) dev: /dev/nvme0n1p3 
Sensors:
  System Temperatures: cpu: 48.0 C mobo: N/A gpu: nvidia temp: 45 C 
  Fan Speeds (RPM): N/A 
Repos:
  Active pacman repo servers in: /etc/pacman.d/mirrorlist 
  1: http://mirror.is.co.za/mirrors/manjaro.org/stable/$repo/$arch
  2: http://manjaro.mirrors.theom.nz/stable/$repo/$arch
  3: https://www.mirrorservice.org/sites/repo.manjaro.org/repos/stable/$repo/$arch
Info:
  Processes: 274 Uptime: 16m Memory: 15.55 GiB used: 1.24 GiB (8.0%) 
  Init: systemd v: 242 Compilers: gcc: 9.2.0 Shell: bash v: 5.0.11 
  running in: xfce4-terminal inxi: 3.0.37

It is already installed (Perhaps I did it wrong somehow)

sudo pip install cchardet
Requirement already satisfied: cchardet in /usr/lib/python3.8/site-packages (2.1.5)

Ok- you must reboot, I forget to write :smiley:

So now the right nvidia driver is installed:

System:
  Kernel: 5.4.12-1-MANJARO x86_64 bits: 64 compiler: gcc v: 9.2.0 
  parameters: BOOT_IMAGE=/boot/vmlinuz-5.4-x86_64 
  root=UUID=7db98f53-a54f-4c21-b2e4-9fff71a772c8 rw quiet apparmor=1 
  security=apparmor resume=UUID=a3d0a02e-e68c-46d1-a9cd-3d271526bf34 
  udev.log_priority=3 
  Desktop: Xfce 4.14.2 tk: Gtk 3.24.13 info: xfce4-panel wm: xfwm4 
  dm: LightDM 1.30.0 Distro: Manjaro Linux 
Machine:
  Type: Laptop System: LENOVO product: 81UH v: Lenovo Legion Y740-15IRHg 
  serial: <filter> Chassis: type: 10 v: Lenovo Legion Y740-15IRHg 
  serial: <filter> 
  Mobo: LENOVO model: LNVNB161216 v: No DPK serial: <filter> UEFI: LENOVO 
  v: BVCN10WW(V1.06) date: 06/06/2019 
Battery:
  ID-1: BAT1 charge: 24.1 Wh condition: 54.7/57.0 Wh (96%) volts: 12.1/11.5 
  model: 0000313100504D53 324750334D37314C type: Li-ion serial: <filter> 
  status: Charging 
CPU:
  Topology: 6-Core model: Intel Core i7-9750H bits: 64 type: MT MCP 
  arch: Kaby Lake family: 6 model-id: 9E (158) stepping: A (10) 
  microcode: CA L2 cache: 12.0 MiB 
  flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx 
  bogomips: 62431 
  Speed: 900 MHz min/max: 800/4500 MHz Core speeds (MHz): 1: 900 2: 900 
  3: 900 4: 900 5: 900 6: 900 7: 900 8: 900 9: 900 10: 902 11: 900 12: 900 
  Vulnerabilities: Type: itlb_multihit status: KVM: Split huge pages 
  Type: l1tf 
  mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable 
  Type: mds mitigation: Clear CPU buffers; SMT vulnerable 
  Type: meltdown mitigation: PTI 
  Type: spec_store_bypass 
  mitigation: Speculative Store Bypass disabled via prctl and seccomp 
  Type: spectre_v1 
  mitigation: usercopy/swapgs barriers and __user pointer sanitization 
  Type: spectre_v2 mitigation: Full generic retpoline, IBPB: conditional, 
  IBRS_FW, STIBP: conditional, RSB filling 
  Type: tsx_async_abort status: Not affected 
Graphics:
  Device-1: NVIDIA TU106BM [GeForce RTX 2060 Mobile] vendor: Lenovo 
  driver: nvidia v: 435.21 bus ID: 01:00.0 chip ID: 10de:1f51 
  Display: x11 server: X.Org 1.20.7 driver: nvidia 
  resolution: 1920x1080~144Hz 
  OpenGL: renderer: GeForce RTX 2060/PCIe/SSE2 v: 4.6.0 NVIDIA 435.21 
  direct render: Yes 
Audio:
  Device-1: Intel Cannon Lake PCH cAVS vendor: Lenovo driver: snd_hda_intel 
  v: kernel bus ID: 00:1f.3 chip ID: 8086:a348 
  Device-2: NVIDIA TU106 High Definition Audio vendor: Lenovo 
  driver: snd_hda_intel v: kernel bus ID: 01:00.1 chip ID: 10de:10f9 
  Sound Server: ALSA v: k5.4.12-1-MANJARO 
Network:
  Device-1: Intel Wireless-AC 9560 [Jefferson Peak] vendor: Bigfoot Networks 
  driver: iwlwifi v: kernel bus ID: 00:14.3 chip ID: 8086:a370 
  IF: wlp0s20f3 state: up mac: <filter> 
  Device-2: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet 
  vendor: Lenovo driver: r8169 v: kernel port: 3000 bus ID: 3f:00.0 
  chip ID: 10ec:8168 
  IF: enp63s0 state: down mac: <filter> 
Drives:
  Local Storage: total: 1.84 TiB used: 7.62 GiB (0.4%) 
  ID-1: /dev/nvme0n1 vendor: Samsung model: MZVLB1T0HALR-000L2 
  size: 953.87 GiB block size: physical: 512 B logical: 512 B 
  speed: 31.6 Gb/s lanes: 4 serial: <filter> rev: 5L1QEXA7 scheme: GPT 
  ID-2: /dev/sda vendor: Seagate model: ST1000LM049-2GH172 size: 931.51 GiB 
  block size: physical: 4096 B logical: 512 B speed: 6.0 Gb/s 
  rotation: 7200 rpm serial: <filter> rev: LXM3 
Partition:
  ID-1: / raw size: 936.46 GiB size: 920.77 GiB (98.32%) 
  used: 7.62 GiB (0.8%) fs: ext4 dev: /dev/nvme0n1p2 
  ID-2: swap-1 size: 17.11 GiB used: 0 KiB (0.0%) fs: swap 
  swappiness: 60 (default) cache pressure: 100 (default) dev: /dev/nvme0n1p3 
Sensors:
  System Temperatures: cpu: 55.0 C mobo: N/A gpu: nvidia temp: 55 C 
  Fan Speeds (RPM): N/A 
Repos:
  Active pacman repo servers in: /etc/pacman.d/mirrorlist 
  1: http://mirror.is.co.za/mirrors/manjaro.org/stable/$repo/$arch
  2: http://manjaro.mirrors.theom.nz/stable/$repo/$arch
  3: https://www.mirrorservice.org/sites/repo.manjaro.org/repos/stable/$repo/$arch
Info:
  Processes: 371 Uptime: N/A Memory: 15.55 GiB used: 993.5 MiB (6.2%) 
  Init: systemd v: 242 Compilers: gcc: 9.2.0 Shell: bash v: 5.0.11 
  running in: xfce4-terminal inxi: 3.0.37

But it still takes forever to run.. :slightly_frowning_face:

Could it be that even though cchardet is installed, for some reason BeautifulSoup isn't using it? (I've also uninstalled chardet- as they recommended on the stackoverflow post from above)

:man_shrugging:

You try also 440?
Kernel 5.5rc7?
New Hardware need time in Linux world for best/better driver software.

https://www.notebookcheck.com/Lenovo-Legion-Y740-15IRHg-81UH001XGE.438494.0.html

80%: This rating is not earth-shattering. Although it is a "satisfactory" rating for most websites, you have to keep in mind that there are hardly any notebooks that receive less than 60%. Verbal ratings in this area often do not sound bad, but are often covert euphemisms. This notebook is actually considered below average by the testers, a purchase recommendation can only be seen with a lot of goodwill in it.

Translated with www.DeepL.com/Translator (free version)

First, NEVER EVER use sudo to pip install ...
Second, @Wollie told you to install the python-cchardet package.

2 Likes

Forum kindly sponsored by