[fedora-arm] Exactly how slow is Fedora 27 on an RP3 ? dnf update takes hours ?

[linux guy]

Hi people.

Thanks to the powers that be for getting Fedora onto the ARM platforms. I
really appreciate your work.

I'm wondering if Fedora 27 is really, really slow on a RPi3 or if I'm doing
something wrong or ???

I installed F27 Server Arm on an RPi3. The file I used was
Fedora-Server-armhfp-27-1.6-sda.raw.xz from https://arm.fedoraproject.org/
via https://download.fedoraproject.org/p...1.6-sda.raw.xz
<https://download.fedoraproject.org/pub/fedora/linux/releases/27/Server/armhfp/images/Fedora-Server-armhfp-27-1.6-sda.raw.xz>
The installation was mostly straight forward.

Naturally the first thing I do is run #dnf update. Upon pressing return
the screen does nothing for 15 minutes. Eventually it comes back saying
~225 packages need updating, 2 need deleting and ~10 need installing. I
say yes to this. The package downloads go OK, takes probably 10 minutes.
But the Upgrading itself is horribly slow. Like 5 minutes per package, on
average.

Is this normal or is there something wrong with my RPi3 ? I thought it
would be faster than this ?

The only thing I can think of is that I'm running it from a 2A power
adapter, rather than the 2.5A that is suggested. It seems to run fine
though, no errors, etc.

Is my power adapter the problem ?

Thanks

[linux guy]

So I checked a few things.

My SD card was a Kingston 16 GB Class 10, rated for 45MB/sec. It should be
OK.

The power adapter was a Samsung unit rated at 2A at 5V. I replaced it
with a unit rated at 2.4A at 5.25V.

I rewrote the SD card. I used gparted to resize the root partition to be
the remainder of the space on the card. It was 3.2GB with 2.3 GB used.
Now it is ~12GB with 2.3 GB used. I wanted to resize the swap partition,
which is set to 488 MB. For some reason gparted wouldn't let me touch it.

I rebooted and reinitialized things. It runs much, much faster, but it
still seems slow. Each package upgrade in #dnf upgrade is probably taking
a minute now, versus 5 minutes before.

[linux guy]

Update.

The dnf update process is still very, very slow on the RPi3. It will take
all night (8 hours) to run. For comparison, I installed F27 workstation
on a Celeron N3000 machine with 2 GB of RAM and a 128 GB SSD earlier this
week. A much larger (workstation versus minimal server) dnf update took in
the order of 10 minutes on that machine. I don't think the I/0 speed and
processing power difference is an order of 1000 between these two
machines.

I ran $top in another console while #dnf update was running. dnf is
rarely at the top of the top listing. And when it is, it is using ~40% of
the CPU and only 12% of the memory. None of the swap memory is being used
at all. 90% of the time $top itself is the largest resource user.

So why is the #dnf update process so slow ? Is it I/O bound ? Is it an
I/O access algorithm problem ?

Post by linux guy
So I checked a few things.
My SD card was a Kingston 16 GB Class 10, rated for 45MB/sec. It should
be OK.
The power adapter was a Samsung unit rated at 2A at 5V. I replaced it
with a unit rated at 2.4A at 5.25V.
I rewrote the SD card. I used gparted to resize the root partition to be
the remainder of the space on the card. It was 3.2GB with 2.3 GB used.
Now it is ~12GB with 2.3 GB used. I wanted to resize the swap partition,
which is set to 488 MB. For some reason gparted wouldn't let me touch it.
I rebooted and reinitialized things. It runs much, much faster, but it
still seems slow. Each package upgrade in #dnf upgrade is probably taking
a minute now, versus 5 minutes before.

[marcin steć]

Seems like my problems with F25 a year ago, so here is what I did, sorry
for lack of details (I'll access the HW within next couple of hours) - SD
Card does matter: I used some budget Sandisk (class 10 I believe) worked
really slow. I moved to some Samsung Evo (class 10 as well - much better).
I also observerd an improvement when I changed io scheduler to noop. Also
an upgrade from 25 to 26 improved a lot. Now hearing You have problems with
27 I'm i doubt... I was about to upgrade this week.
When I was investigating this problem with nmon I've foud that
systemd-journald is responsible for "heavy" IO. Does it perform IO
synchronously ?

Post by linux guy
Update.
The dnf update process is still very, very slow on the RPi3. It will take
all night (8 hours) to run. For comparison, I installed F27 workstation
on a Celeron N3000 machine with 2 GB of RAM and a 128 GB SSD earlier this
week. A much larger (workstation versus minimal server) dnf update took in
the order of 10 minutes on that machine. I don't think the I/0 speed and
processing power difference is an order of 1000 between these two
machines.
I ran $top in another console while #dnf update was running. dnf is
rarely at the top of the top listing. And when it is, it is using ~40% of
the CPU and only 12% of the memory. None of the swap memory is being used
at all. 90% of the time $top itself is the largest resource user.
So why is the #dnf update process so slow ? Is it I/O bound ? Is it an
I/O access algorithm problem ?

Post by linux guy
So I checked a few things.
My SD card was a Kingston 16 GB Class 10, rated for 45MB/sec. It should
be OK.
The power adapter was a Samsung unit rated at 2A at 5V. I replaced it
with a unit rated at 2.4A at 5.25V.
I rewrote the SD card. I used gparted to resize the root partition to be
the remainder of the space on the card. It was 3.2GB with 2.3 GB used.
Now it is ~12GB with 2.3 GB used. I wanted to resize the swap partition,
which is set to 488 MB. For some reason gparted wouldn't let me touch it.
I rebooted and reinitialized things. It runs much, much faster, but it
still seems slow. Each package upgrade in #dnf upgrade is probably taking
a minute now, versus 5 minutes before.

_______________________________________________

[linux guy]

Thanks for the reply.

The update just completed. It took 8 hours. The transaction required
Installing 32 packages, upgrading 275 packages and skipping 2 packages.
Ironically, a new update just became available as I write this !

I took pictures of the transaction and top screens with my phone if anyone
wants to see them.

I ask that someone test updating a fresh install on an RPi3 to see what
they get.

Marcin: how much faster was the Evo card than the Sandisk card ? How much
difference did using noop make ? I need my RPi3 to be an order of
magnitude faster.

Thanks
​

[Nigel Sollars]

Ive used Kingston SD cards ( class 10's ) in many SBC's and have no speed
issues. Im currently using GSkill class 10's, but I think your safe to
take the SD card off the table ( unless your specifically getting i/o
errors in syslog / dmesg? ).

I would be searching your logs for potential hardware failure as what you
describe is far from normal.

Nige

Post by linux guy
Thanks for the reply.
The update just completed. It took 8 hours. The transaction required
Installing 32 packages, upgrading 275 packages and skipping 2 packages.
Ironically, a new update just became available as I write this !
I took pictures of the transaction and top screens with my phone if anyone
wants to see them.
I ask that someone test updating a fresh install on an RPi3 to see what
they get.
Marcin: how much faster was the Evo card than the Sandisk card ? How
much difference did using noop make ? I need my RPi3 to be an order of
magnitude faster.
Thanks
​
_______________________________________________

[Peter Robinson]

Post by linux guy
Thanks for the reply.
The update just completed. It took 8 hours. The transaction required
Installing 32 packages, upgrading 275 packages and skipping 2 packages.
Ironically, a new update just became available as I write this !
I took pictures of the transaction and top screens with my phone if anyone
wants to see them.
I ask that someone test updating a fresh install on an RPi3 to see what they
get.

It's not the fastest in the world but I've not seen it that slow.

I use either Samsung EVO or Sandisk cards, I've had problems with
Kingston in the past so don't waste my time with them any more.

dnf itself will be slow in the beginning because of the requirements
to process a lot of XML data and that is slow. We're working to remove
this from client side, that will take time.

We've improved mmc speed a lot from the original supported release in
F-25, as someone said F-26 was better and F-27 is certainly no slower
than prior releases.

There should be improved speed for MMC in general in 4.16 which will
be the GA kernel for F-28. I'm hoping we'll have some other speed
improvements in F-28 around the RPi as well but that will depend on my
time available to work on some of that functionality.

Peter
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-le

[Richard Ryniker]

It is definitely a good idea to check the system journal. If something
wrong is recorded there, there are likely many instances logged to cause
hours of unexpected delay.

I find it better to use ssh to connect to my RPi than to suffer its slow
graphics performance. Time information for a 'dnf upgrade" today on my
RPi (that included installation of the 4.14.14-300.fc27 kernel) appears
below.

After the upgrade, a second run (with nothing to upgrade) takes 18 to 19
seconds of elapsed time.

I use a 32GB Samsung EVO memory card in my RPi. This data tells you what
I experience, to compare with your own system.
________________________________________________________________________________

[***@rpi3-1 ryniker]# time dnf upgrade
Last metadata expiration check: 0:46:01 ago on Wed Jan 24 08:10:39 2018.
Dependencies resolved.
...
Installing dependencies:
gdbm-devel armv7hl 1:1.13-6.fc27 updates 63 k
Removing:
kernel armv7hl 4.14.8-300.fc27 @updates 0
kernel-core armv7hl 4.14.8-300.fc27 @updates 89 M
kernel-modules armv7hl 4.14.8-300.fc27 @updates 17 M

Transaction Summary
================================================================================
Install 4 Packages
Upgrade 85 Packages
Remove 3 Packages

Total download size: 189 M
...
Complete!

real 19m33.591s
user 10m28.092s
sys 1m58.729s
[***@rpi3-1 ryniker]#
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-leave

[linux guy]

I just ran #time dnf install rygel on my fresh install. It required 56
packages.

real 64m43.986s
user 1m59.778s
sys 0m25.893s

This is on a console only machine, no GUI.

Could someone run the same process on their RPi3 and see what they get ?

​

[linux guy]

Found the issue:

sudo hdparm -Tt /dev/mmcblk0



/dev/mmcblk0:

Timing cached reads: 240 MB in 2.00 seconds = 119.84 MB/sec

Timing buffered disk reads: 2 MB in 5.60 seconds = 365.82 kB/sec
<------ Wow !


Bad memory card ????

Eh, I’ll give it a go
 on F26 armhf7l. Keep in mind my repos may be
different, so it might very the timing by maybe 30s?
time sudo dnf install rygel –y
Unfortunately it’s only 21 packages for me. And now that I look at it,
would probably be better to “dnf download” first and time that separately.
//start snip


Complete!
real 3m34.645s
user 1m54.094s
sys 1m19.396s
# and
sudo hdparm -Tt /dev/mmcblk0
Timing cached reads: 834 MB in 2.00 seconds = 417.21 MB/sec
Timing buffered disk reads: 68 MB in 3.06 seconds = 22.24 MB/sec
//end snip
Thanks.
-Bryan
*Sent:* Wednesday, January 24, 2018 2:31 PM
*To:* Richard Ryniker
*Subject:* [fedora-arm] Re: Exactly how slow is Fedora 27 on an RP3 ? dnf
update takes hours ?
I just ran #time dnf install rygel on my fresh install. It required 56
packages.
real 64m43.986s
user 1m59.778s
sys 0m25.893s
This is on a console only machine, no GUI.
Could someone run the same process on their RPi3 and see what they get ?
​
This communication may contain information that is proprietary,
confidential, or exempt from disclosure. If you are not the intended
recipient, please note that any other dissemination, distribution, use or
copying of this communication is strictly prohibited. Anyone who receives
this message in error should notify the sender immediately by telephone or
by return e-mail and delete it from his or her computer.

[linux guy]

I removed the suspect SD card from the RPi3 and tested it in my laptop.

# hdparm -Tt /dev/mmcblk0

/dev/mmcblk0:
Timing cached reads: 15750 MB in 2.00 seconds = 7893.74 MB/sec
Timing buffered disk reads: 88 MB in 3.05 seconds = 28.82 MB/sec <---
This was 365 kB/sec in the RPi3.

It seems to test OK in the laptop. Bad connection in the RPi3 ?

So I prepared a second class 10 SD card with the same image. I tested it in
the laptop that I used to prepare it.

# hdparm -Tt /dev/mmcblk0

/dev/mmcblk0:
Timing cached reads: 15908 MB in 2.00 seconds = 7973.32 MB/sec
Timing buffered disk reads: 54 MB in 3.05 seconds = 17.71 MB/sec

The second SD is actually slower to read than the first one, in the
laptop. 28.82 versus 17.71 MB/sec

I booted the 2nd SD card in the RPi3.

# hdparm -Tt /dev/mmcblk0

/dev/mmcblk0:
Timing cached reads: 816 MB in 2.00 seconds 408.21 MB/sec
Timing buffered disk reads: 66 MB in 3.03 seconds = 21.81 MB/sec <---
Faster than it was in the laptop !

#time dnf update - install 32 packages, update 275, just like last time

I ran #time dnf update and stopped it at 47 packages.

real: 30m46.181s
user: 0.00s
sys: 0m.020s

I retested the SD card and got the same read speeds as above.

That would be a good place to start. FWIW, all my micro SD cards are
Sandisk ultra’s . Unfortunately I don’t see that /sys/block is well
populated (no model).
-Bryan
*Sent:* Wednesday, January 24, 2018 2:51 PM
*To:* Duff, Bryan
*Subject:* Re: [fedora-arm] Re: Exactly how slow is Fedora 27 on an RP3 ?
dnf update takes hours ?
sudo hdparm -Tt /dev/mmcblk0
Timing cached reads: 240 MB in 2.00 seconds = 119.84 MB/sec
Timing buffered disk reads: 2 MB in 5.60 seconds = 365.82 kB/sec
<------ Wow !
Bad memory card ????
Eh, I’ll give it a go
 on F26 armhf7l. Keep in mind my repos may be
different, so it might very the timing by maybe 30s?
time sudo dnf install rygel –y
Unfortunately it’s only 21 packages for me. And now that I look at it,
would probably be better to “dnf download” first and time that separately.
//start snip


Complete!
real 3m34.645s
user 1m54.094s
sys 1m19.396s
# and
sudo hdparm -Tt /dev/mmcblk0
Timing cached reads: 834 MB in 2.00 seconds = 417.21 MB/sec
Timing buffered disk reads: 68 MB in 3.06 seconds = 22.24 MB/sec
//end snip
Thanks.
-Bryan
*Sent:* Wednesday, January 24, 2018 2:31 PM
*To:* Richard Ryniker
*Subject:* [fedora-arm] Re: Exactly how slow is Fedora 27 on an RP3 ? dnf
update takes hours ?
I just ran #time dnf install rygel on my fresh install. It required 56
packages.
real 64m43.986s
user 1m59.778s
sys 0m25.893s
This is on a console only machine, no GUI.
Could someone run the same process on their RPi3 and see what they get ?
​
This communication may contain information that is proprietary,
confidential, or exempt from disclosure. If you are not the intended
recipient, please note that any other dissemination, distribution, use or
copying of this communication is strictly prohibited. Anyone who receives
this message in error should notify the sender immediately by telephone or
by return e-mail and delete it from his or her computer.

[linux guy]

Update/solved.

So I finally got back to work on this project. I did installed another
image on the Kingston card and ran dnf update. It took > 8 hours.

On the advice from my friend I purchased a Class 3 SanDisk "Extreme" 32 GB
micro SD card. The packaging claims reads at 90 MB/sec (600x) and writes
at 60 MB/sec (400x).

# time dnf update with this card now yields: real 37m8s, user 27m39s sys
4m36s. At least 16x faster. And the current dnf update is larger than
what I was running several weeks ago.

#dnf install rygel takes a 5 minutes instead of over an hour.

Bottom line: the SD card (speed) makes a huge difference in the speed of an
RPi3.

Thanks for the help.
​

[Peter Robinson]

Post by linux guy
Update/solved.
So I finally got back to work on this project. I did installed another
image on the Kingston card and ran dnf update. It took > 8 hours.
On the advice from my friend I purchased a Class 3 SanDisk "Extreme" 32 GB
micro SD card. The packaging claims reads at 90 MB/sec (600x) and writes at
60 MB/sec (400x).
# time dnf update with this card now yields: real 37m8s, user 27m39s sys
4m36s. At least 16x faster. And the current dnf update is larger than
what I was running several weeks ago.
#dnf install rygel takes a 5 minutes instead of over an hour.
Bottom line: the SD card (speed) makes a huge difference in the speed of an
RPi3.

Yes, I tend to use either the Sandisk Extreme or Samsung EVO lines of SD card.

There's also been a big rework of the MMC subsystem that landed in
4.16 that moved to the block MQ kernel infra which should help
performances, and will also enabled the use of HW queues on supported
HW eventually so there should be improvements seen in F-28 for SD card
performance.

Peter
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.fed

[Stephen John Smoogen]

Post by linux guy
Update.
The dnf update process is still very, very slow on the RPi3. It will take
all night (8 hours) to run. For comparison, I installed F27 workstation on
a Celeron N3000 machine with 2 GB of RAM and a 128 GB SSD earlier this week.
A much larger (workstation versus minimal server) dnf update took in the
order of 10 minutes on that machine. I don't think the I/0 speed and
processing power difference is an order of 1000 between these two machines.
I ran $top in another console while #dnf update was running. dnf is rarely
at the top of the top listing. And when it is, it is using ~40% of the CPU
and only 12% of the memory. None of the swap memory is being used at all.
90% of the time $top itself is the largest resource user.

That would say it is not anything to do with the CPU and something to
do with the IO. The first thing I would look at is that the Kingston
card may have been bootlegged or it is shoddy. There were a large
number of fake cards out there where someone relabeled lower quality
ones to higher ones. The second is that the speed of IO is going to be
peak read/write rates and not average rates. This means that some
manufacturer cards will be rated as EV10 but only when they are
running downhill with the wind to their back (as they would say about
cars).

Finally the connector to the rasp pi to the MMC may be dirty. The old
clean with alchohol and such have helped on other models for other
people. That said, MMC is slow. It is not as fast as SSD in any shape
or form so that even an old slower celeron is going to smoke the
raspberry pi nearly every time.



--
Stephen J Smoogen.
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email