[fedora-arm] NanoPi Neo2, Rawhide-20180330: (probably) hangs in initial-setup

[Tomasz Kłoczko]

Hi,

Whole boot process is OK until:

[ OK ] Mounted /boot.
Mounting /boot/efi...
[ OK ] Mounted /boot/efi.
[ OK ] Reached target Local File Systems.
Starting Restore /run/initramfs on shutdown...
Starting Import network configuration from initramfs...
Starting Rebuild Journal Catalog...
Starting Rebuild Dynamic Linker Cache...
Starting Tell Plymouth To Write Out Runtime Data...
[ OK [ 39.871989] audit: type=1130 audit(1520306229.519:93): pid=1
uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0
msg='unit=dracut-shutdown c'
] Started Restore /run/initramfs on shutdown.
[ OK ] Started Tell Plymouth To Write Out Runtime Data.
[ OK ] Started Rebuild Journal Catalog.
[ OK ] Started Import network configuration from initramfs.
[ OK ] Started Rebuild Dynamic Linker Cache.
Starting Update is Completed...
Starting Create Volatile Files and Directories...
[ OK ] Started Update is Completed.
[ OK ] Started Create Volatile Files and Directories.
Starting Security Auditing Service...
[ OK ] Started Security Auditing Service.
Starting Update UTMP about System Boot/Shutdown...
[ OK ] Started Update UTMP about System Boot/Shutdown.
[ OK ] Reached target System Initialization.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Listening on SSSD Secrets Service responder socket.
[ OK ] Started dnf makecache timer.
[ OK ] Reached target Timers.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Listening on Open-iSCSI iscsiuio Socket.
[ OK ] Listening on Open-iSCSI iscsid Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting Login Service...
[ OK ] Started D-Bus System Message Bus.
[ OK ] Started Hardware RNG Entropy Gatherer Daemon.
Starting Initial Setup configuration program...
[ OK ] Started Secure Boot DBX (blacklist) updater.
Starting NTP client/server...
[ OK ] Reached target sshd-keygen.target.
Starting firewalld - dynamic firewall daemon...

On some boots is displayed FAIL on NTP and firewalld services but
except this nothing more happens.

Suggestions/comments?

kloczek
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.fedoraproj

[Peter Robinson]

Post by Tomasz Kłoczko
Hi,
[ OK ] Mounted /boot.
Mounting /boot/efi...
[ OK ] Mounted /boot/efi.
[ OK ] Reached target Local File Systems.
Starting Restore /run/initramfs on shutdown...
Starting Import network configuration from initramfs...
Starting Rebuild Journal Catalog...
Starting Rebuild Dynamic Linker Cache...
Starting Tell Plymouth To Write Out Runtime Data...
[ OK [ 39.871989] audit: type=1130 audit(1520306229.519:93): pid=1
uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0
msg='unit=dracut-shutdown c'
] Started Restore /run/initramfs on shutdown.
[ OK ] Started Tell Plymouth To Write Out Runtime Data.
[ OK ] Started Rebuild Journal Catalog.
[ OK ] Started Import network configuration from initramfs.
[ OK ] Started Rebuild Dynamic Linker Cache.
Starting Update is Completed...
Starting Create Volatile Files and Directories...
[ OK ] Started Update is Completed.
[ OK ] Started Create Volatile Files and Directories.
Starting Security Auditing Service...
[ OK ] Started Security Auditing Service.
Starting Update UTMP about System Boot/Shutdown...
[ OK ] Started Update UTMP about System Boot/Shutdown.
[ OK ] Reached target System Initialization.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Listening on SSSD Secrets Service responder socket.
[ OK ] Started dnf makecache timer.
[ OK ] Reached target Timers.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Listening on Open-iSCSI iscsiuio Socket.
[ OK ] Listening on Open-iSCSI iscsid Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting Login Service...
[ OK ] Started D-Bus System Message Bus.
[ OK ] Started Hardware RNG Entropy Gatherer Daemon.
Starting Initial Setup configuration program...
[ OK ] Started Secure Boot DBX (blacklist) updater.
Starting NTP client/server...
[ OK ] Reached target sshd-keygen.target.
Starting firewalld - dynamic firewall daemon...
On some boots is displayed FAIL on NTP and firewalld services but
except this nothing more happens.
Suggestions/comments?

Do you have a serial console?

kloczek

Post by Tomasz Kłoczko
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
_______________________________________________

[Tomasz Kłoczko]

On 1 April 2018 at 03:58, Peter Robinson <***@gmail.com> wrote:
[..]

Post by Peter Robinson
Do you have a serial console?

Maybe it is not obvious but NanoPi Neo2 has *only* serial console and
what I'v sent it is output fron serial console.

kloczek
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.fedoraproject

[Peter Robinson]

Post by Tomasz Kłoczko
[..]

Post by Peter Robinson
Do you have a serial console?

Maybe it is not obvious but NanoPi Neo2 has *only* serial console and

I've had reports of output on HDMI and then it disappearing even
though the device has continued to boot so you're correct it's not
obvious and you need to be explicit in bug reports.
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscr

[Tomasz Kłoczko]

On 1 April 2018 at 04:20, Peter Robinson <***@gmail.com> wrote:
[..]

Post by Peter Robinson
I've had reports of output on HDMI and then it disappearing even
though the device has continued to boot so you're correct it's not
obvious and you need to be explicit in bug reports.

Probably it was other NanoPi. This is how NanoPi Neo2 looks like:
http://www.friendlyarm.com/index.php?route=product/product&product_id=180

So .. do you have any suggestions about what I can do to
diagnose/debug this issue?

This output which I've copied was after I found that ssk-keygen
services has been frozen as well.
I've generated necessary keys b mounting root fs on my laptop and
generate those keys using ssh-keygen commands.
Initially I've been thinking that just generate those keys takes to long.
Now to be hones I think that freezing ssh-keygen services and
initial-setup may have te same cause.
However none of those services have been failing after some time by
timeout which is strange.
Why I think that it is strange? Because IMO default systemd settings
should be failing if service cannot finish in some time defined
globally or per service.
It is only explanation why everything waits in infinite loop.

Any suggestions about change systemd settings to fail service if it
will not start within let's say 60s?

kloczek
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.fed

[Tomasz Kłoczko]

On 1 April 2018 at 04:37, Tomasz Kłoczko <***@gmail.com> wrote:
[..]
So I've left my NnoPi powered and ond the morning I found on console:

[ 4463.943146] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4463.956184] RTL8211E Gigabit Ethernet 0.2:07: attached PHY driver
[RTL8211E Gigabit Ethernet] (mii_bus:phy_addr=0.2:07, irq=POLL)
[ 4464.004639] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management
Counters available
[ 4464.012846] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW
[ 4464.027487] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 4516.416930] DMA-API: cacheline tracking ENOMEM, dma-debug disabled
[ 4516.423543] DMA-API: cacheline tracking ENOMEM, dma-debug disabled
[ 4536.767953] python3 (1214) used greatest stack depth: 9632 bytes left
[ 4536.774894] python3 (1202) used greatest stack depth: 9520 bytes left
[ OK ] Started Initial Setup configuration program.
[ OK ] Reached target Login Prompts (Pre).
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Started Getty on tty1.
[ OK ] Reached target Login Prompts.

Fedora 29 (Rawhide)
Kernel 4.16.0-0.rc7.git1.1.fc29.aarch64 on an aarch64 (ttyS0)

localhost login: [ 8796.686735] IPv6: ADDRCONF(NETDEV_UP): eth0: link
is not ready
[13038.537106] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[13129.146222] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[17821.059949] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

After plugging eth cable:

[21666.647917] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[FAILED] Failed to start Network Manager.
See 'systemctl status NetworkManager.service' for details.
[ OK ] Stopped Serial Getty on ttyS0.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Started D-Bus System Message Bus.
[ OK ] Stopped OpenSSH server daemon.
[ OK ] Stopped target sshd-keygen.target.
Stopping sshd-keygen.target.
[ OK ] Reached target sshd-keygen.target.
Starting OpenSSH server daemon...
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.
[FAILED] Failed to start OpenSSH server daemon.
See 'systemctl status sshd.service' for details.
[ OK ] Stopped Login Service.
Starting Login Service...
[ OK ] Stopped Network Manager.
[ OK ] Started D-Bus System Message Bus.
Starting Network Manager...

Fedora 29 (Rawhide)
Kernel 4.16.0-0.rc7.git1.1.fc29.aarch64 on an aarch64 (ttyS0)

localhost login:

So looks like I was right that everything was not instantly because
initial-setup has been doing something.
I've been trying to login but looks like root has some password.


After reboot at the end of boot sequence is:

[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
[ OK ] Started D-Bus System Message Bus.
Starting NTP client/server...
[ OK ] Started Hardware RNG Entropy Gatherer Daemon.
Starting firewalld - dynamic firewall daemon...
Starting Login Service...
[ OK ] Started Secure Boot DBX (blacklist) updater.
[ OK ] Reached target sshd-keygen.target.
[FAILED] Failed to start NTP client/server.
See 'systemctl status chronyd.service' for details.
[FAILED] Failed to start firewalld - dynamic firewall daemon.
See 'systemctl status firewalld.service' for details.
[ OK ] Started D-Bus System Message Bus.
[ OK ] Reached target Network (Pre).
Starting Network Manager...
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.
[ OK ] Stopped Login Service.
Starting Login Service...
[ OK ] Started D-Bus System Message Bus.
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.
[ OK ] Stopped Login Service.
Starting Login Service...
[ OK ] Started D-Bus System Message Bus.
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.
[ OK ] Stopped Login Service.
Starting Login Service...
[FAILED] Failed to start Network Manager.
See 'systemctl status NetworkManager.service' for details.
[ OK ] Started D-Bus System Message Bus.
[ OK ] Reached target Network.
[ OK ] Started Logout off all iSCSI sessions on shutdown.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
Starting Permit User Sessions...
Starting OpenSSH server daemon...
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.
[ OK ] Stopped Login Service.
Starting Login Service...
[ OK ] Stopped Network Manager.
[ OK ] Started D-Bus System Message Bus.
Starting Network Manager...
[ OK ] Started Permit User Sessions.
Starting Hold until boot process finishes up...
Starting Terminate Plymouth Boot Screen...
[FAILED] Failed to start OpenSSH server daemon.
See 'systemctl status sshd.service' for details.
[ OK ] Started Hold until boot process finishes up.
[ OK ] Started Terminate Plymouth Boot Screen.

Fedora 29 (Rawhide)
Kernel 4.16.0-0.rc7.git1.1.fc29.aarch64 on an aarch64 (ttyS0)

localhost login:


So still something went wrong.

kloczek
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send

[Tomasz Kłoczko]

On 1 April 2018 at 12:02, Tomasz Kłoczko <***@gmail.com> wrote:
]..]
I've repeat test today with today
Fedora-Minimal-Rawhide-20180401.n.0.aarch64.raw.xz image.

Here is end part of the boot log on serial console:

[ OK ] Started Rebuild Dynamic Linker Cache.
Starting Update is Completed...
[ OK ] Started Update is Completed.
[ OK ] Reached target System Initialization.
[ OK ] Listening on Open-iSCSI iscsid Socket.
[ OK ] Listening on SSSD Secrets Service responder socket.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Started dnf makecache timer.
[ OK ] Reached target Timers.
[ OK ] Listening on Open-iSCSI iscsiuio Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting OpenSSH ecdsa Server Key Generation...
Starting Login Service...
[ OK ] Started D-Bus System Message Bus.
Starting Initial Setup configuration program...
[ OK ] Started Secure Boot DBX (blacklist) updater.
Starting OpenSSH ed25519 Server Key Generation...
[ OK ] Started Hardware RNG Entropy Gatherer Daemon.
Starting firewalld - dynamic firewall daemon...
Starting NTP client/server...
Starting OpenSSH rsa Server Key Generation...
[FAILED] Failed to start OpenSSH ecdsa Server Key Generation.
See 'systemctl status sshd-***@ecdsa.service' for details.
[FAILED] Failed to start OpenSSH ed25519 Server Key Generation.
See 'systemctl status sshd-***@ed25519.service' for details.
[FAILED] Failed to start OpenSSH rsa Server Key Generation.
See 'systemctl status sshd-***@rsa.service' for details.

After those fails I've left NanoPi Neo2 running with connected eth cable.
After about an hour next part of the console logs appeared:

[ 3573.381667] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 3573.392157] RTL8211E Gigabit Ethernet 0.2:07: attached PHY driver
[RTL8211E Gigabit Ethernet] (mii_bus:phy_addr=0.2:07, irq=POLL)
[ 3573.419960] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management
Counters available
[ 3573.427816] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW
[ 3573.439070] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 3576.559307] dwmac-sun8i 1c30000.ethernet eth0: Link is Up -
100Mbps/Full - flow control off
[ 3576.568165] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 3753.881940] DMA-API: cacheline tracking ENOMEM, dma-debug disabled
[ 3753.888624] DMA-API: cacheline tracking ENOMEM, dma-debug disabled
[ 4242.259031] python3 (1410) used greatest stack depth: 9632 bytes left
[ OK ] Started D-Bus System Message Bus.
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.
[FAILED] Failed to start Network Manager.
See 'systemctl status NetworkManager.service' for details.
[ OK ] Stopped Login Service.
Starting Login Service...
[ OK ] Started Initial Setup configuration program.
[ OK ] Stopped Network Manager.
Starting Network Manager...
[ OK ] Reached target Login Prompts (Pre).
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Started Getty on tty1.
[ OK ] Reached target Login Prompts.

Fedora 29 (Rawhide)
Kernel 4.16.0-0.rc7.git1.1.fc29.aarch64 on an aarch64 (ttyS0)

localhost login: root
Password:
Login incorrect

Despite that I've wrote image on the sd card with --norootpass I was
not able to login.
I've left NanoPi running and started doing other things and after
about another hour next batch of boot log messages on SC appeared:

[ OK ] Started D-Bus System Message Bus.
[ OK ] Stopped OpenSSH server daemon.
[ OK ] Stopped target sshd-keygen.target.
Stopping sshd-keygen.target.
Starting OpenSSH ecdsa Server Key Generation...
Starting OpenSSH rsa Server Key Generation...
Starting OpenSSH ed25519 Server Key Generation...
[FAILED] Failed to start Network Manager.
See 'systemctl status NetworkManager.service' for details.
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.
[FAILED] Failed to start OpenSSH ecdsa Server Key Generation.
See 'systemctl status sshd-***@ecdsa.service' for details.
[FAILED] Failed to start OpenSSH rsa Server Key Generation.
See 'systemctl status sshd-***@rsa.service' for details.
[FAILED] Failed to start OpenSSH ed25519 Server Key Generation.
See 'systemctl status sshd-***@ed25519.service' for details.
[ OK ] Stopped Network Manager.
[ OK ] Started D-Bus System Message Bus.
Starting Network Manager...
[ OK ] Stopped Login Service.
Starting Login Service...
[ OK ] Stopped Serial Getty on ttyS0.
[ OK ] Started Serial Getty on ttyS0.
[ OK ] Reached target sshd-keygen.target.
Starting OpenSSH server daemon...
[ OK ] Started D-Bus System Message Bus.
[FAILED] Failed to start OpenSSH server daemon.
See 'systemctl status sshd.service' for details.
[ OK ] Started D-Bus System Message Bus.

Fedora 29 (Rawhide)
Kernel 4.16.0-0.rc7.git1.1.fc29.aarch64 on an aarch64 (ttyS0)

localhost login: root
Password:
Login incorrect

So:
1) looks like first try to generate ssh keys fails for some still
unknown reasons
2) --norootpass and --addkey options are not working (I was not able
to login over ssh when ssh at least started).

Here is full command line which I've used to write image:

# fedora-arm-image-installer
--image=Fedora-Minimal-Rawhide-20180401.n.0.aarch64.raw.xz
--media=/dev/sdd --target=nanopi_neo2 --norootpass -y --addconsole
--addkey=id_rsa.pub

Will try latter to have look for some logs left on the image after initial boot.
I'll try lated as well with today image and RPi3.
Looking as well for any other suggestions about what can I do more to
diagnose those issues.

kloczek

[Tomasz Kłoczko]

On 1 April 2018 at 20:36, Tomasz Kłoczko <***@gmail.com> wrote:
[..]
OK here are few more things which I fund:
1) initial-setup and ssh key generation services do not work because
something is really screwed with SElinux settings and when SELinux has
enabled those services seems are not able to write keys or apply other
modifications.

With those two modifications so far I've been able to finish first boot with:

[ OK ] Started Secure Boot DBX (blacklist) updater.
[ OK ] Started Hardware RNG Entropy Gatherer Daemon.
Starting OpenSSH ecdsa Server Key Generation...
Starting Initial Setup configuration program...
Starting OpenSSH ed25519 Server Key Generation...
[ 51.795341] bridge: filtering via arp/ip/ip6tables is no longer
available by default. Update your scripts to load br_netfilter if you
need this.
[ 51.922063] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 51.933399] RTL8211E Gigabit Ethernet 0.2:07: attached PHY driver
[RTL8211E Gigabit Ethernet] (mii_bus:phy_addr=0.2:07, irq=POLL)
[ 51.974155] dwmac-sun8i 1c30000.ethernet eth0: No MAC Management
Counters available
[ 51.982348] dwmac-sun8i 1c30000.ethernet eth0: PTP not supported by HW
[ 51.994992] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 55.119566] dwmac-sun8i 1c30000.ethernet eth0: Link is Up -
100Mbps/Full - flow control off
[ 55.128530] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
================================================================================
================================================================================
1) [x] Language settings 2) [x] Time settings
(English (United States)) (US/Eastern timezone)
3) [x] Network configuration 4) [x] Root password
(Wired (eth0) connected) (Password is set.)
5) [ ] User creation
(No user will be created)

Please make a selection from the above ['c' to continue, 'q' to quit,
'r' to refresh]:


2) --norootpass option does not work because in used to generate image
ks profiles is:

rootpw --iscrypted --lock locked

so in /etc/shadow is line:

root:!locked::0:99999:7:::

when in fedora-arm-image-installer is line

sed -i 's/root:x:/root::/' /tmp/root/etc/passwd

Obviously, this is not putting empty root password and root account
still is locked.
Despite this seems this option can be removed now because anaconda
started in initial-setup service provides root password change.

I think that fedora-arm-image-installer script should be evolving now
into the direction tweaking ks profile used on first boot and/or force
as well start anaconda in non-interactive mode.
Probably the best way to fix SELinux issues is to change SELInux
settings to disabled (in ks profile used for generating images).
Seems that hostname changes are not applied because the system should
be rebooted after finish anaconda.
If in used in initial-setup ks profile would enable SELinux and before
reboot will be created /.autorelabel file all still incorrect SELinux
labels will be corrected on next reboot.

kloczek

[Peter Robinson]

Post by Tomasz Kłoczko
[..]
1) initial-setup and ssh key generation services do not work because
something is really screwed with SElinux settings and when SELinux has
enabled those services seems are not able to write keys or apply other
modifications.

Should be fixed in GA
I only see one.

Post by Tomasz Kłoczko
2) --norootpass option does not work because in used to generate image
rootpw --iscrypted --lock locked
when in fedora-arm-image-installer is line
sed -i 's/root:x:/root::/' /tmp/root/etc/passwd

Patches welcome.

Post by Tomasz Kłoczko
Obviously, this is not putting empty root password and root account
still is locked.
Despite this seems this option can be removed now because anaconda
started in initial-setup service provides root password change.

There are other use cases other than just that.

Post by Tomasz Kłoczko
I think that fedora-arm-image-installer script should be evolving now
into the direction tweaking ks profile used on first boot and/or force
as well start anaconda in non-interactive mode.

I don't know what that provides, please link to documentation

Post by Tomasz Kłoczko
Probably the best way to fix SELinux issues is to change SELInux
settings to disabled (in ks profile used for generating images).

That won't ever happen, SELinux is a key security feature and if
there's bugs we file them and get them fixed.

Post by Tomasz Kłoczko
Seems that hostname changes are not applied because the system should
be rebooted after finish anaconda.

In most cases when the host name is changed you need to reboot to make
sure all things referencing the hostname are updated, there's nothing
new or special here.

Post by Tomasz Kłoczko
If in used in initial-setup ks profile would enable SELinux and before
reboot will be created /.autorelabel file all still incorrect SELinux
labels will be corrected on next reboot.

Or the bugs are reported and fixed for the next image creation.
_______________________________________________
arm mailing list -- ***@lists.fedoraproject.org
To unsubscribe send an email to arm-***@lists.fedorapro