[fedora-arm] Qemu aarch64 install

[Sean Omalley]

I tried to set up an aarch64 vm, and not really wanting to set up boot server and such. I just ran.


virt-install \
--name Fedora_25_AArch64 --ram 2048 --arch aarch64 \
--disk size=8 --os-variant fedora25 \
--location https://dl.fedoraproject.org/pub/fedora-secondary/releases/25/Everything/aarch64/os/
--extra-args "inst.ks=https://pwhalen.fedorapeople.org/kickstarts/Fedora-Minimal-AArch64.ks"

It almost worked. I already had qemu-kvm set up for a windows partition for my vehicle obdII software probably with group install virtualization, then I installed qemu-system-arm and qemu-system-aarch64 along with edk2-aarch64 libguestfs-tools-c

It bombed out at (I have the full install log including the backtrace but I didn't want to spam the list with it.) It appeared to be going okay for a while. It is using the 4.8.6-300 kernel, is there an SMP bug in it? Or any suggestions?


Downloading 302 RPMs, 194.8 MiB / 194.8 MiB (100%) done.
Preparing transaction from installation source
[ 3151.714055] Internal error: Oops - BUG: 0 [#1] SMP
[ 3151.717402] Modules linked in: vfat fat fcoe libfcoe libfc scsi_transport_fc zram crc32_arm64 ghash_ce mtd chipreg qemu_fw_cfg loop virtio_scsi virtio_console virtio_net gpio_keys virtio_mmio virtio_ring virtio sunrpc xts lrw gf128mul dm_crypt dm_round_robin linear raid10 raid456 async_raid6_recov async_memcpy libcrc32c async_pq async_xor xor async_tx raid6_pq raid1 raid0 scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua iscsi_tcp libiscsi_tcp libiscsi squashfs cramfs scsi_transport_iscsi dm_multipath
[ 3151.724909] CPU: 0 PID: 1228 Comm: loop0 Tainted: P 4.8.6-300.fc25.aarch64 #1
[ 3151.749929] Hardware name: QEMU QEMU Virtual Machine, BIOS 0.0.0 02/06/2015
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.

[Gerd Hoffmann]

Post by Sean Omalley
I tried to set up an aarch64 vm, and not really wanting to set up boot server and such. I just ran.
virt-install \
--name Fedora_25_AArch64 --ram 2048 --arch aarch64 \
--disk size=8 --os-variant fedora25 \
--location https://dl.fedoraproject.org/pub/fedora-secondary/releases/25/Everything/aarch64/os/
--extra-args "inst.ks=https://pwhalen.fedorapeople.org/kickstarts/Fedora-Minimal-AArch64.ks"
It almost worked. I already had qemu-kvm set up for a windows partition for my vehicle obdII software probably with group install virtualization, then I installed qemu-system-arm and qemu-system-aarch64 along with edk2-aarch64 libguestfs-tools-c
It bombed out at (I have the full install log including the backtrace but I didn't want to spam the list with it.) It appeared to be going okay for a while. It is using the 4.8.6-300 kernel, is there an SMP bug in it? Or any suggestions?

Had problems too, although I just saw the install hang on post-install
steps. Installed f24 instead, worked. Then fedup to f25, worked too.

My guess is that the f25 install media kernel is bad. With fedup the
latest f25 kernel from updates is used instead.

cheers,
Gerd
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an ema

[Paul Whalen]

----- Original Message -----

Post by Sean Omalley
I tried to set up an aarch64 vm, and not really wanting to set up boot server
and such. I just ran.
virt-install \
--name Fedora_25_AArch64 --ram 2048 --arch aarch64 \
--disk size=8 --os-variant fedora25 \
--location
https://dl.fedoraproject.org/pub/fedora-secondary/releases/25/Everything/aarch64/os/
--extra-args
"inst.ks=https://pwhalen.fedorapeople.org/kickstarts/Fedora-Minimal-AArch64.ks"
It almost worked. I already had qemu-kvm set up for a windows partition for
my vehicle obdII software probably with group install virtualization, then I
installed qemu-system-arm and qemu-system-aarch64 along with edk2-aarch64
libguestfs-tools-c
It bombed out at (I have the full install log including the backtrace but I
didn't want to spam the list with it.) It appeared to be going okay for a
while. It is using the 4.8.6-300 kernel, is there an SMP bug in it? Or any
suggestions?

Unfortunately you need allocate more ram, I generally use 4096, but you might
be able to get by with a little less. Using 2048 I can reproduce the same result.

The wiki does use 4096 in the example, but I'll make a note of the crash as
well.

Paul

Post by Sean Omalley
Downloading 302 RPMs, 194.8 MiB / 194.8 MiB (100%) done.
Preparing transaction from installation source
[ 3151.714055] Internal error: Oops - BUG: 0 [#1] SMP
[ 3151.717402] Modules linked in: vfat fat fcoe libfcoe libfc
scsi_transport_fc zram crc32_arm64 ghash_ce mtd chipreg qemu_fw_cfg loop
virtio_scsi virtio_console virtio_net gpio_keys virtio_mmio virtio_ring
virtio sunrpc xts lrw gf128mul dm_crypt dm_round_robin linear raid10 raid456
async_raid6_recov async_memcpy libcrc32c async_pq async_xor xor async_tx
raid6_pq raid1 raid0 scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua
iscsi_tcp libiscsi_tcp libiscsi squashfs cramfs scsi_transport_iscsi
dm_multipath
[ 3151.724909] CPU: 0 PID: 1228 Comm: loop0 Tainted: P
4.8.6-300.fc25.aarch64 #1
[ 3151.749929] Hardware name: QEMU QEMU Virtual Machine, BIOS 0.0.0 02/06/2015
_______________________________________________

_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.

[Sean Omalley]

----- Original Message -----
From: Paul Whalen <***@redhat.com>
To: Sean Omalley <***@rocketmail.com>
Cc: ***@lists.fedoraproject.org
Sent: Wednesday, February 8, 2017 10:48 AM
Subject: Re: [fedora-arm] Qemu aarch64 install



----- Original Message -----

Post by Sean Omalley
I tried to set up an aarch64 vm, and not really wanting to set up boot server
and such. I just ran.
virt-install \
--name Fedora_25_AArch64 --ram 2048 --arch aarch64 \
--disk size=8 --os-variant fedora25 \
--location
https://dl.fedoraproject.org/pub/fedora-secondary/releases/25/Everything/aarch64/os/
--extra-args
"inst.ks=https://pwhalen.fedorapeople.org/kickstarts/Fedora-Minimal-AArch64.ks"
It almost worked. I already had qemu-kvm set up for a windows partition for
my vehicle obdII software probably with group install virtualization, then I
installed qemu-system-arm and qemu-system-aarch64 along with edk2-aarch64
libguestfs-tools-c
It bombed out at (I have the full install log including the backtrace but I
didn't want to spam the list with it.) It appeared to be going okay for a
while. It is using the 4.8.6-300 kernel, is there an SMP bug in it? Or any
suggestions?

Unfortunately you need allocate more ram, I generally use 4096, but you might
be able to get by with a little less. Using 2048 I can reproduce the same result.

The wiki does use 4096 in the example, but I'll make a note of the crash as
well.


---
Thanks! I will try that!


It stopped exactly in the same place the second time I tried. I added --vcpus 1.


When I tried through the Virt-manager gui and tried an http install it got quite a bit further. It installed 312/485, then a similar error, then continued and finally died at 408/485 with a server install, I wasn't sure how to add the anaconda file so it was more packages.


I figured it created a swapfile if it needed more ram. It was the same settings I gave win10 and what I typically give to linux x86 guests.


Sean
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an em

[Paul Whalen]

Post by Paul Whalen
----- Original Message -----
Sent: Wednesday, February 8, 2017 10:48 AM
Subject: Re: [fedora-arm] Qemu aarch64 install
----- Original Message -----

Post by Sean Omalley
I tried to set up an aarch64 vm, and not really wanting to set up boot server
and such. I just ran.
virt-install \
--name Fedora_25_AArch64 --ram 2048 --arch aarch64 \
--disk size=8 --os-variant fedora25 \
--location
https://dl.fedoraproject.org/pub/fedora-secondary/releases/25/Everything/aarch64/os/
--extra-args
"inst.ks=https://pwhalen.fedorapeople.org/kickstarts/Fedora-Minimal-AArch64.ks"
It almost worked. I already had qemu-kvm set up for a windows partition for
my vehicle obdII software probably with group install virtualization, then I
installed qemu-system-arm and qemu-system-aarch64 along with edk2-aarch64
libguestfs-tools-c
It bombed out at (I have the full install log including the backtrace but I
didn't want to spam the list with it.) It appeared to be going okay for a
while. It is using the 4.8.6-300 kernel, is there an SMP bug in it? Or any
suggestions?

Unfortunately you need allocate more ram, I generally use 4096, but you might
be able to get by with a little less. Using 2048 I can reproduce the same result.
The wiki does use 4096 in the example, but I'll make a note of the crash as
well.
---
Thanks! I will try that!
It stopped exactly in the same place the second time I tried. I added --vcpus 1.
When I tried through the Virt-manager gui and tried an http install it got
quite a bit further. It installed 312/485, then a similar error, then
continued and finally died at 408/485 with a server install, I wasn't sure
how to add the anaconda file so it was more packages.
I figured it created a swapfile if it needed more ram. It was the same
settings I gave win10 and what I typically give to linux x86 guests.

We lose some ram on aarch64 due to cma allocation, its something that is
actively being looked at. Anaconda also needs a fair amount of ram during
the network install to copy files. Once the install is complete, you should
be able to reduce the ram to 2048.

Paul

Post by Paul Whalen
Sean

_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to

[Gerd Hoffmann]

Hi,

Post by Paul Whalen
We lose some ram on aarch64 due to cma allocation, its something that is
actively being looked at. Anaconda also needs a fair amount of ram during
the network install to copy files. Once the install is complete, you should
be able to reduce the ram to 2048.

2G not being enough is pretty excessive. I never had such problems on
x86. Only recently (a year ago or so) I started using 2G for guests, 1G
used to be enough even for network installs.

I suspect a big factor here is CONFIG_ARM64_64K_PAGES=y. Raspberry pi
3, with standard fedora kernel (64k pages), minimal install (@core),
booted to the shell prompt. "free" reports ~122MB of memory as used.
Same with a self-compiled kernel (4k pages), I get ~44MB of used memory.
That is almost factor three!

cheers,
Gerd

_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send a

[Peter Robinson]

Post by Gerd Hoffmann

Post by Paul Whalen
We lose some ram on aarch64 due to cma allocation, its something that is
actively being looked at. Anaconda also needs a fair amount of ram during
the network install to copy files. Once the install is complete, you should
be able to reduce the ram to 2048.

2G not being enough is pretty excessive. I never had such problems on
x86. Only recently (a year ago or so) I started using 2G for guests, 1G
used to be enough even for network installs.

Completely agree, generally if you're doing a anaconda based install
you need at least 768Mb due to various combinations of installer
memory usage and selinux applying policies.

Post by Gerd Hoffmann
I suspect a big factor here is CONFIG_ARM64_64K_PAGES=y. Raspberry pi
booted to the shell prompt. "free" reports ~122MB of memory as used.
Same with a self-compiled kernel (4k pages), I get ~44MB of used memory.
That is almost factor three!

That is almost the complete factor. It's in fact a combination of 64K
pages and CMA. The two together mean that the minimum CMA allocated is
512Mb of RAM. It's a known problem and one we're looking at different
ways/options to resolve and working with upstream to do so and
reconsidering 64K pages in Fedora. We've traditionally gone that route
due to large enterprise systems and to be in sync with downstreams
like RHELSA and CentOS in that regard.

Peter
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe

[Pavel Holica]

Post by Paul Whalen

Post by Gerd Hoffmann

Post by Paul Whalen
We lose some ram on aarch64 due to cma allocation, its something that is
actively being looked at. Anaconda also needs a fair amount of ram

during

Post by Gerd Hoffmann

Post by Paul Whalen
the network install to copy files. Once the install is complete, you

should

Post by Gerd Hoffmann

Post by Paul Whalen
be able to reduce the ram to 2048.

2G not being enough is pretty excessive. I never had such problems on
x86. Only recently (a year ago or so) I started using 2G for guests, 1G
used to be enough even for network installs.

Completely agree, generally if you're doing a anaconda based install
you need at least 768Mb due to various combinations of installer
memory usage and selinux applying policies.

That's true. If you're looking for smallest memory usage during
installation possible, use NFS installation source (or ISO or ISO over
NFS). That way, install.img (anaconda rootfs - 378MiB large in case of F25)
is mounted directly from nfs shares and packages are also used directly
from NFS, no copying to local / (which is using memory through
device-mapper). HTTP installation consumes the most memory, since it's
copying install.img and other stuff to local / (again in memory). Afaik
RHEL-7 has 1G requirement for NFS installation and 2G for HTTP
installation. The last thing you want to happen is to have OOM during
installation :)

Post by Paul Whalen

Post by Gerd Hoffmann
I suspect a big factor here is CONFIG_ARM64_64K_PAGES=y. Raspberry pi
booted to the shell prompt. "free" reports ~122MB of memory as used.
Same with a self-compiled kernel (4k pages), I get ~44MB of used memory.
That is almost factor three!

That is almost the complete factor. It's in fact a combination of 64K
pages and CMA. The two together mean that the minimum CMA allocated is
512Mb of RAM. It's a known problem and one we're looking at different
ways/options to resolve and working with upstream to do so and
reconsidering 64K pages in Fedora. We've traditionally gone that route
due to large enterprise systems and to be in sync with downstreams
like RHELSA and CentOS in that regard.
Peter
_______________________________________________

[Gerd Hoffmann]

Hi,

Post by Peter Robinson

Post by Gerd Hoffmann
I suspect a big factor here is CONFIG_ARM64_64K_PAGES=y. Raspberry pi
booted to the shell prompt. "free" reports ~122MB of memory as used.
Same with a self-compiled kernel (4k pages), I get ~44MB of used memory.
That is almost factor three!

That is almost the complete factor. It's in fact a combination of 64K
pages and CMA. The two together mean that the minimum CMA allocated is
512Mb of RAM. It's a known problem and one we're looking at different
ways/options to resolve and working with upstream to do so and
reconsidering 64K pages in Fedora. We've traditionally gone that route
due to large enterprise systems and to be in sync with downstreams
like RHELSA and CentOS in that regard.

What was the reason to go 64k pages in the first place? Sure, with
larger pages memory management overhead goes down. But on the other
hand the memory footprint goes up, and frankly I'm a bit surprised how
much it goes up. So I'm wondering whenever 64k pages is a net win even
on enterprise machines. Did people benchmark this?

cheers,
Gerd
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.fedoraproje

[Peter Robinson]

Post by Gerd Hoffmann

Post by Peter Robinson

Post by Gerd Hoffmann
I suspect a big factor here is CONFIG_ARM64_64K_PAGES=y. Raspberry pi
booted to the shell prompt. "free" reports ~122MB of memory as used.
Same with a self-compiled kernel (4k pages), I get ~44MB of used memory.
That is almost factor three!

That is almost the complete factor. It's in fact a combination of 64K
pages and CMA. The two together mean that the minimum CMA allocated is
512Mb of RAM. It's a known problem and one we're looking at different
ways/options to resolve and working with upstream to do so and
reconsidering 64K pages in Fedora. We've traditionally gone that route
due to large enterprise systems and to be in sync with downstreams
like RHELSA and CentOS in that regard.

What was the reason to go 64k pages in the first place? Sure, with
larger pages memory management overhead goes down. But on the other
hand the memory footprint goes up, and frankly I'm a bit surprised how
much it goes up. So I'm wondering whenever 64k pages is a net win even
on enterprise machines. Did people benchmark this?

TBH I'm not sure of all of the reasons, for some use cases like
hypervisors where the processes you have running (VMs) tend to have
large memory allocations I can see it's benefits.
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists

[Jon Masters]

On 02/16/2017 04:10 AM, Gerd Hoffmann wrote:

<snip>

Post by Gerd Hoffmann
What was the reason to go 64k pages in the first place?

Several reasons:

1). Some (server) implementations have higher performance under 64K.
2). VA sizes greater than 48-bit require a 64K translation granule. I
can mention that one publicly now. There are similar reasons in the
pipeline for why 64K is going to be needed on big iron ARM systems.
3). We thought 64K was going to matter, and we wanted to make sure we
could support it (4K is "easy", those other guys do that already).

Post by Gerd Hoffmann
Sure, with larger pages memory management overhead goes down. But on
the other hand the memory footprint goes up, and frankly I'm a bit
surprised how much it goes up.

There's much room for optimization. I want to try to avoid throwing the
baby out with the bathwater as we push ARM and others to clean this up.
For another example, we see kernel structure ballooning caused by the
lack of support for sparse CPU masks and the like - all things that
ARM should be addressing in upstream. Our using 64K helps to keep the
pressure on them to clean this up. RHEL (and Cent) will use 64K no
matter what, but there could be a (short term) case for Fedora having
a cycle or two with a smaller size - I would prefer to avoid that.

Post by Gerd Hoffmann
So I'm wondering whenever 64k pages is a net win even
on enterprise machines. Did people benchmark this?

It is a net win on Enterprise, required for some of the insanely large
machines being designed now. I can give you one example - Cray have
recently announced that they will be shipping a very large ARM system
in the next couple months. There are many other such machines coming.

Benchmarking was done based upon models by the RH perf team about
3-4 years ago, yes. Again, there's a lot of cleanup to do, but that's
not in and of itself a reason to throw out 64K, especially as it's the
only path to >48-bit VA support in the coming iteration.

Jon.

[Gerd Hoffmann]

Hi,

Post by Jon Masters
pressure on them to clean this up. RHEL (and Cent) will use 64K no
matter what, but there could be a (short term) case for Fedora having
a cycle or two with a smaller size - I would prefer to avoid that.

I think it would be more useful to offer a 4k kernel in addition to the
64k one, similar to how Fedora ships kernel with and without lpae
support on armv7.

I expect on low-end / hobbyist devices like the raspberry pi 3 with only
1G ram a 64k kernel is pretty pointless IMO.

cheers,
Gerd
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an

[Jon Masters]

Post by Gerd Hoffmann

Post by Jon Masters
pressure on them to clean this up. RHEL (and Cent) will use 64K no
matter what, but there could be a (short term) case for Fedora having
a cycle or two with a smaller size - I would prefer to avoid that.

I think it would be more useful to offer a 4k kernel in addition to the
64k one, similar to how Fedora ships kernel with and without lpae
support on armv7.

I suggested this before. I won't print what the response was :)

However, if you want to bring it up with the kernel@ team and drive...

Jon.

[Marcin Juszkiewicz]

Post by Gerd Hoffmann
I think it would be more useful to offer a 4k kernel in addition to
the 64k one, similar to how Fedora ships kernel with and without
lpae support on armv7.

Some applications still check PAGE_SIZE only during compilation instead
of on runtime. Btrfs was also hardcoding it.
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to

[Gerd Hoffmann]

Post by Marcin Juszkiewicz

Post by Gerd Hoffmann
I think it would be more useful to offer a 4k kernel in addition to
the 64k one, similar to how Fedora ships kernel with and without
lpae support on armv7.

Some applications still check PAGE_SIZE only during compilation instead
of on runtime. Btrfs was also hardcoding it.

Can't be too bad. I'm running f25 aarch64 userspace with a 4k pages
kernel on the rpi3. The only problem I ran into so far is that the
swapspace format depends on the pagesize too, so you have to run "swapon
--fixpgsz" once after switching from 64k to 4k or visa versa.

cheers,
Gerd
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.fedoraproject.org

[Jon Masters]

Suggestion: if you're somehow able to persuade folks to do a 4K kernel in addition, retain default 64K. Packages built for 64K will mostly be on with 4K, but not the inverse.
--
Computer Architect | Sent from my 64-bit #ARM Powered phone

Post by Gerd Hoffmann

Post by Marcin Juszkiewicz

Post by Gerd Hoffmann
I think it would be more useful to offer a 4k kernel in addition to
the 64k one, similar to how Fedora ships kernel with and without
lpae support on armv7.

Some applications still check PAGE_SIZE only during compilation instead
of on runtime. Btrfs was also hardcoding it.

Can't be too bad. I'm running f25 aarch64 userspace with a 4k pages
kernel on the rpi3. The only problem I ran into so far is that the
swapspace format depends on the pagesize too, so you have to run "swapon
--fixpgsz" once after switching from 64k to 4k or visa versa.
cheers,
Gerd
_______________________________________________

_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe s

[sean o]

_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.fedoraproject.org