Decode a MikroTik autosupout.rif yourself: 20 lines of Python found the out-of-memory kernel panic

A MikroTik router kept rebooting every 10 minutes to 2.6 hours, with almost no traffic. After each boot, the log said:

system,error,critical router was rebooted without proper shutdown, probably kernel failure
system,error,critical kernel failure in previous boot
system,error,critical out of memory condition was detected
system,error,critical Automatic supout.rif file generated due to service malfunction, please contact MikroTik support and supply the generated file

You don’t have to wait for support. autosupout.rif is base64 with the bits in the other order, wrapped around zlib, and about 20 lines of Python turn it into plain text: the kernel’s last crash dumps, the log, /proc and the full config. That’s where the cause was:

Out of memory and no killable processes...
Kernel panic - not syncing: System is deadlocked on memory

The setup #

• MikroTik hAP ac² (IPQ4019, 128 MiB RAM, 16 MiB flash)
• RouterOS 6.49.5, then 7.23.8 (long-term) with the legacy wireless package; both crashed the same way
• Python 3 on a Mac to decode the file

The file format #

Copy the file off the router and look at it:

scp admin@192.168.0.1:/autosupout.rif .
file autosupout.rif
head -2 autosupout.rif

autosupout.rif: ASCII text, with CRLF line terminators
--BEGIN ROUTEROS SUPOUT SECTION
uQWZiV3ZAgHnt3X72tNOSb+7xXFYO5HTy0WSEgfKeG3rdsd643J2xjtT37s5kVHKSIbOWiUNJVs9

It looks like base64, and base64 -d turns it into noise. The alphabet is standard, but each group of four characters is read as a 24-bit little-endian number: the first character holds the lowest 6 bits, and the first output byte is the lowest 8 bits. Decoded, each section is its name, a NUL byte, and a zlib stream.

The decoder #

import os, re, sys, zlib

ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'

def decode(text):
    text = re.sub(r'[^A-Za-z0-9+/=]', '', text)
    out = bytearray()
    for i in range(0, len(text), 4):
        group = text[i:i + 4].rstrip('=')
        value = sum(ALPHABET.index(c) << (6 * j) for j, c in enumerate(group))
        out += value.to_bytes(3, 'little')[:len(group) * 6 // 8]
    return bytes(out)

src, outdir = sys.argv[1], sys.argv[2]
os.makedirs(outdir, exist_ok=True)
sections = re.findall(r'--BEGIN ROUTEROS SUPOUT SECTION\r?\n(.*?)--END ROUTEROS SUPOUT SECTION',
                      open(src, encoding='ascii').read(), re.S)
for n, section in enumerate(sections):
    name, _, data = decode(section).partition(b'\0')
    text = zlib.decompressobj().decompress(data)
    safe = re.sub(r'[^\w.-]', '_', name.decode(errors='replace'))
    with open(os.path.join(outdir, f'{n:02d}_{safe}.txt'), 'wb') as f:
        f.write(text)
    print(f'{n:02d} {safe} {len(text)}')

python3 supout.py autosupout.rif supout/

The 6.49.5 file had 76 sections and the 7.23.8 one 87. The useful ones:

What it showed #

On 7.23.8, every crash in oops ended the same way, after a process table in which every process had the same score:

[  691.378048][  T142] [    150]     0   150     1667      153    28672        0         -1000 parser
...
[  691.378492][  T142] Out of memory and no killable processes...
[  691.378509][  T142] Kernel panic - not syncing: System is deadlocked on memory

RouterOS runs every process with oom_score_adj -1000, so the OOM killer has nothing to kill and the kernel panics. Any out-of-memory, however brief, is a reboot. 6.49.5 got there by another route, with the kernel set to panic on any OOM: Kernel panic - not syncing: Out of memory: compulsory panic_on_oom is enabled.

The processes weren’t the problem, at about 0.5 MiB each. resource, written a minute after boot, was:

MemTotal:         118248 kB
MemFree:            7368 kB
SUnreclaim:        52756 kB

One minute into the boot, the kernel already held 52 MiB of unreclaimable memory. The caches in slabinfo add up to only about 10 MiB, so the rest is allocated outside them. In two of the panics, other CPUs were inside the Wi-Fi driver (ath_poll, __w_qos_decrypt_mpdu in the wlan and ath modules), which makes it the main suspect, but turning one radio off didn’t settle it.

The same reading works on any embedded Linux box: check oom_score_adj in the OOM dump, then SUnreclaim against what the process table and slabinfo explain.

Read the timestamps with care #

The router has no battery-backed clock. After a crash, it restores an old time, and NTP takes minutes to catch up, so log and supout timestamps right after a reboot were about 7 minutes early. Five “crashes” stamped within two seconds of each other were most likely one boot loop.

The supout itself was written a minute after the crash. The e-mail service tried to send the crash notice before PPPoE and DNS were up, died, and that service failure is what produced the file (.debug shows error: DNS resolve failed [code: 21] and then died).

Before you send it to anyone #

The decoded sections held, in plain text, the PPPoE password, the WireGuard private key, a TSIG key, and a bearer token from a dynamic DNS script. Rotate those before you hand the file to MikroTik support or attach it to a forum post.

On a router this tight, delete the file after you copy it. The root of the file system is in RAM, and a 480 KiB supout is RAM the kernel needed:

/file remove [find name~"autosupout"]

Takeaway #

A supout.rif holds the kernel log, /proc and your config, and you can read it in a minute. Next time your MikroTik writes one, decode it and read oops before you open a support ticket.