The NetBSD 9.3 and OpenBSD 7.2 (and upcoming FreeBSD 14.0) releases include one small fix to the establishment of RDC R6040 (vte(4)) Ethernet controller’s link on DM&P Vortex86DX3 dual-core SoC. More than that, the similar patch was applied on Linux kernel as well, making it my first contribution to all major BSDs and Linux. It is only few lines in the code, but it was a long journey to identify it. Thus, I decided to write a small article with the hope to encourage people to work on such issues.

Back in 2018, I bought a small DM&P Vortex86DX3 SoC based system, called eBox 3352DX3-AP. It is USB powered fanless system, which includes few I/O ports: 3xUSB 2.0 ports, 1xVGA (Vortex86VGA), 1xSD Card slot and one 100Mbit Ethernet RJ-45 port (RDC R6040 MAC). AP abbreviation means auto-power, which indicates that it doesn’t have physical power button and starts automatically as soon as power cable is connected to it. NetBSD was, of course, the first OS I’ve tried to boot on it. Unfortunately, it wasn’t seamless experience, the two major issues were clear on the initial attempts:

• ACPI/SMP needed to be disabled, otherwise USB (and later identified that all other devices (audio/network), using legacy PCI INTx interrupts) will fail to work. This issue is still unresolved, and it is NetBSD/OpenBSD specific (any help appreciated!). FreeBSD/Linux does seem to be capable in handling those interrupts without any issues.
• R6040 Ethernet controller failed to work, doesn’t matter if ACPI was enabled or disabled. At the point of discovery I thought that it was BSD specific (affected all of them the same way), but later on it appeared to be the issue in Linux as well, even though 4.x and early 5.x kernel versions worked with the controller due to the way Linux was establishing the connection.

I took a priority on investigating network controller issue with the hope, that it would be easier to solve, and additionally would allow to use SSH to connect to the device on ACPI boot (I didn’t know yet that level interrupts causing the failure, on network controller won’t work without fixing that). More than that, I really hoped to resolve it pretty fast, compared to quite troublesome navigation between USB and/or ACPI/APIC code. Unfortunately, I was quite wrong with this assumption. It took me around 3 years to identify the cause and apply the final patch! Nevertheless, I had a working device much sooner than that, but let’s start story from the beginning .

Initial investigation

The first step is always to check dmesg(8) messages of the system and check ifconfig vte0 command output. On my PR report I stated that PHY OUI was not the same as the driver expected (0xfcff2f vs 0x00d02d) and it had a “new” model (0x0005 instead of 0x0003). Because of that general phy driver (ukphy(4)) was attached instead of rdcphy(4) driver. MAC controller (vte) was recognized correctly and attached as expected though.

vte0 at pci0 dev 8 function 0: vendor 17f3 product 6040 (rev. 0x00)
vte0: Ethernet address xx:xx:xx:xx:xx:xx
vte0: interrupting at irq 5
ukphy0 at vte0 phy 1: OUI 0xfcff2f, model 0x0005, rev. 0
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto

As the first step, I started by adding a “new” oui and model to miidevs file, and included it in rdcphy.c phys list, so it can be recognized and used instead of generic one. Even though rdcphy attached successfully after the changes, the link state remained unset. Disappointed, I took a more careful look into the phy driver code. Main focus was on the MII_MEDIACHG case, and especially rdcphy_status() method. I didn’t find anything wrong with MII_MEDIACHG, but status method had at least 4 conditional statements before it sets media link! That was a really good spot to go deeper. Especially on two specific conditional statements.

Running ifconfig vte0 revealed that link wasn’t established: media: Ethernet autoselect (none). Manual attempts to set the link and its speed were unsuccessful too. This behavior was the same in FreeBSD and OpenBSD. Linux was working though (used SparkyLinux distribution with some 4.x kernel). The connection wasn’t always stable, but it worked. It also gave me a additional false hope to resolve this soon, since I assumed it may be enough to compare drivers and see what was different between them.

	PHY_READ(sc, MII_BMSR, &bmsr);
	PHY_READ(sc, MII_BMSR, &bmsr);
	PHY_READ(sc, MII_RDCPHY_STATUS, &physts);

	if ((physts & STATUS_LINK_UP) != 0)
		mii->mii_media_status |= IFM_ACTIVE;

	PHY_READ(sc, MII_BMCR, &bmcr);
	if ((bmcr & BMCR_ISO) != 0) {
		mii->mii_media_active |= IFM_NONE;
		mii->mii_media_status = 0;
		return;
	}

	if ((bmcr & BMCR_LOOP) != 0)
		mii->mii_media_active |= IFM_LOOP;

	if ((bmcr & BMCR_AUTOEN) != 0) {
		if ((bmsr & BMSR_ACOMP) == 0) {
			/* Erg, still trying, I guess... */
			mii->mii_media_active |= IFM_NONE;
			return;
		}
	}

From the rdcphy(4) code above we can see that bmsr (Basic Mode Status Register) register is being read twice, and may be used later to check if auto-negotiation is still in progress. However, the first check compares MII_RDCPHY_STATUS register value against STATUS_LINK_UP mask to set device active. Finally, bmcr (Basic Mode Control Register) register value is checked against BMCR_ISO mask to decide if driver can proceed or link status can’t be identified. This condition picked my eyes, since ifconfig was showing that media link status is “none”. I printed values of all of these three registers and it appeared that BMCR and MII_RDCPHY_STATUS always reports 0xffff. BMSR value was correct on the other hand. Thus, first patch was to ignore all these if conditions and set media link to IFM_100_TX “by force” in the code. Quite surprisingly, it worked! Network controller established the link and successfully auto-negotiated IP address. Obviously it was not a solution to commit, but it worked as initial workaround and could be applied locally.

Comparison with Linux driver

At this point I started to compare the driver to Linux kernel counterpart, including its various older iterations. Unfortunately, the code functionally was very similar, except few small differences. Of course, those can be vital sometimes, but over the time I tried to match both drivers as close as possible without any positive results. The biggest hopes were associated with this part:

	/* If PHY status change register is still set to zero it means the
	 * bootloader didn't initialize it, so we set it to:
	 * - enable phy status change
	 * - enable all phy addresses
	 * - set to lowest timer divider */
	if (ioread16(ioaddr + PHY_CC) == 0)
		iowrite16(SCEN | PHY_MAX_ADDR << PHYAD_SHIFT |
				7 << TMRDIV_SHIFT, ioaddr + PHY_CC);

The comment above states, if PHY_CC register has value 0, the PHY status change register wasn’t initialized yet by the boot loader. Writing certain value to it would enable phy status change, would enable all PHY addresses and would set to the lowest timer divider. I was so convinced that it was a missing link, that disappointment was really great, once it appeared not to be the case. No matter where I applied this code, it had no effect on resolving the issue. Soon after I started to print all PHY register values, which indicated that every second register address returns 0xffff. And it seems that those were mainly “write” registers, read-only ones were successfully returning values. I was stuck with no solid ideas how to move forward after that.

Next lead – controller reset function

I didn’t give up completely though. I was trying to change or comment various places in the code, especially in device initialization code (vte_init, vte_attach). While doing so, I was still printing registry values and comparing dmesg. During some of those experiments, I noticed that commenting out vte_reset() call suddenly changes the situation, network controller starts to work, rdcphy OUI becomes an expected value, registered for RDC devices (0x00d02d instead of 0xfcff2f). And the most importantly, all PHY registers started to print “correct” values, by which I mean there were not 0xffff anymore! Additional tests showed that calling this function once is enough to make controller to stop working (even more specifically any of CSR_WRITE_2(sc, VTE_MCR1, mcr | MCR1_MAC_RESET) or CSR_WRITE_2(sc, VTE_MACSM, 0x0002); and CSR_WRITE_2(sc, VTE_MACSM, 0); calls would result to reproducible issue). Surprisingly, Linux MAC reset code was doing exactly the same thing(!), leaving me confused on why the code works on this system. It was pretty important breakthrough, which not only allowed to simplify temporary patch to commenting out MAC reset code, but it finally enabled me to focus on much more narrow space. What is more, I could compare register values before and after reset. Initially, I believed that, it might be an issue of the 0x0005 model, but this theory was debunked soon by checking submitted dmesg outputs in dmesgd.nycbug.org (it is really helpful database of dmesg outputs(!)). I quickly found dmesg outputs of the systems with the different DM&P Vortex86 SoC models (not DX3) printing correct OUI value for the same 0x0005 model, meaning vte_reset() wasn’t affecting them in the same way as my system.

At this point, I really wanted to check directly how Linux actually behaves, but I failed to build a boot-able kernel (much later I found out it’s because of too big initrd, but I will come back on that later). Additionally, there was no way to print phy OUI value, it is not available in dmesg, and all mii tools failed to print it. Instead, I managed to acquire a Vortex86DX2 SoC based system and started comparing register values before and after reset between two systems. Even though this comparison was indicating the issue, I failed to recognize it at that time. I saw few registers being different between two before reset and becoming the same after, however I just tried to set DX2 values before reset, which didn’t solve the issue. For quite a long time I was stuck again. I was endlessly reading Linux code (including older version of the driver), trying random patches on NetBSD driver, nevertheless failing to find any possible lead.

Vortex86EX2 and bootable custom Linux kernel

At some point I decided to buy one more Vortex86 based system, this time based on the newest Vortex86EX2 SoC. I didn’t buy it to investigation the issue specifically (had an application for it in my house network), but it gave me an opportunity to compare registry values between three systems now. EX2 was working out of the box, despite having even newer 0x0006 model PHY. The registry values appeared closer to DX2 system than DX3. I contacted ICOP support at that point (with whom I had a pleasant communication regarding other issues in the past), leading to the communication between different people, sister companies and even the original seller of DX3 system. Unfortunately, the final response was “we don’t know, Linux works, must be something wrong with BSD drivers”. It encouraged me to take one more attempt to build custom Linux kernel. This was the point when I realized that system fails to boot because of much bigger initrd file than the original was (thus it was exceeding the size of available RAM in the system). It appeared that I need to strip debug symbols by adding INSTALL_MOD_STRIP=1 property in make modules_install command. And the custom build Linux kernel booted successfully! I started to print registry values around MAC reset calls and, to my big surprise Linux had exactly the same issue on printing 0xffff value for every second register. It was the important turning point, indicating that Linux is equally affected and no magic happens. It took some time to understand by reading Linux code why it worked at all. Discussions with my colleague helped me to realize that I am reading the newest code only, while I should probably check the code of the kernel used by my current distribution. It appeared that previous Linux kernel versions used only BMSR register value to establish link before this commit, and this specific register still works well after reset. However, starting 5.3 kernel version BMCR register value was taken into account as well, making code closer to BSDs and PHY was failing to link up in the same way after that change. Just to be sure, I tested edge Sparky Linux distribution version, which used the newer kernel and confirmed that I couldn’t establish a link with R6040 Ethernet controller anymore without removing reset calls. At this point I almost completely gave up and I even decided to close my bug report, deemed my system to be likely broken.

Final issue identification and fix

Despite the decision to close my the bug report and I didn’t completely gave up on the debugging the issue. I was still pretty sure that it is something with registers and I decided yet again to look at the differences between my DX3 and DX2/EX2 systems. I re-identified two registers, whose would reset to different value and at least one of them would reset to the value original DX2/EX2 value before reset. DX2/EX2 register value wouldn’t change on the hand after reset. One of those registers was MDC (Management Data Input/Output Interface Clock) speed control register (defined as VTE_MDCSC in NetBSD). As previously, I attempted to set the DX2 value before the reset and nothing would change, or so it seemed. However, this time I decided to add more logging and print all register values not only before and after the reset, but before and after setting affected register value. It was this decision which allowed me to notice that registers start to go “crazy” not after reset action anymore, but after setting this register’s value to one after reset! After that, the fix was on the horizon, I just needed to set original register value before reset back after the MAC reset. I researched more on what this register do, and once I found out that it drives the clock by the MAC device to PHY, everything made sense to me. The default register value have been already defined in the NetBSD too (#define MDCSC_DEFAULT 0x0030). Thus the final fix had only few additional lines of code in MAC reset function:

static void
vte_reset(struct vte_softc *sc)
{
	uint16_t mcr;
	uint16_t mcr, mdcsc;
	int i;

	mdcsc = CSR_READ_2(sc, VTE_MDCSC);
	mcr = CSR_READ_2(sc, VTE_MCR1);
	CSR_WRITE_2(sc, VTE_MCR1, mcr | MCR1_MAC_RESET);
	for (i = VTE_RESET_TIMEOUT; i > 0; i--) {
@@ -1231,6 +1232,14 @@ vte_reset(struct vte_softc *sc)
	CSR_WRITE_2(sc, VTE_MACSM, 0x0002);
	CSR_WRITE_2(sc, VTE_MACSM, 0);
	DELAY(5000);

	/*
	 * On some SoCs (like Vortex86DX3) MDC speed control register value
	 * needs to be restored to original value instead of default one,
	 * otherwise some PHY registers may fail to be read.
	 */
	if (mdcsc != MDCSC_DEFAULT)
		CSR_WRITE_2(sc, VTE_MDCSC, mdcsc);
}

It reads VTE_MDCSC value before reset and stores it to mdcsc variable. After the MAC reset, mdcsc value is compared to the default register value (0x0030), and being set to the original, if it is not equal to default. If it is the same as default, there is no need to set the same value again.

The bug report was reported on 3rd of August, 2018, the fix was committed on 30th of August, 2021. It took quite a lots of time and effort to discover, but it was really rewarding and educational, especially when it was eventually applied to all the major BSDs and Linux kernel code.

Unfortunately, network controller still doesn’t work on ACPI/SMP mode on NetBSD/OpenBSD, because of level interrupts failure, which makes USB/audio to fail as well, and it is another issue I need to identify and fix.