Mikrotik on Lessons learned

Decode a MikroTik autosupout.rif yourself: 20 lines of Python found the out-of-memory kernel panic

Wed, 14 Oct 2026 18:17:00 +0200

A MikroTik router kept rebooting every 10 minutes to 2.6 hours, with almost no traffic. After each boot, the log said:

system,error,critical router was rebooted without proper shutdown, probably kernel failure
system,error,critical kernel failure in previous boot
system,error,critical out of memory condition was detected
system,error,critical Automatic supout.rif file generated due to service malfunction, please contact MikroTik support and supply the generated file

You don’t have to wait for support. autosupout.rif is base64 with the bits in the other order, wrapped around zlib, and about 20 lines of Python turn it into plain text: the kernel’s last crash dumps, the log, /proc and the full config. That’s where the cause was:

Out of memory and no killable processes...
Kernel panic - not syncing: System is deadlocked on memory

The setup

MikroTik hAP ac² (IPQ4019, 128 MiB RAM, 16 MiB flash)
RouterOS 6.49.5, then 7.23.8 (long-term) with the legacy wireless package; both crashed the same way
Python 3 on a Mac to decode the file

The file format

Copy the file off the router and look at it:

scp admin@192.168.0.1:/autosupout.rif .
file autosupout.rif
head -2 autosupout.rif

autosupout.rif: ASCII text, with CRLF line terminators
--BEGIN ROUTEROS SUPOUT SECTION
uQWZiV3ZAgHnt3X72tNOSb+7xXFYO5HTy0WSEgfKeG3rdsd643J2xjtT37s5kVHKSIbOWiUNJVs9

It looks like base64, and base64 -d turns it into noise. The alphabet is standard, but each group of four characters is read as a 24-bit little-endian number: the first character holds the lowest 6 bits, and the first output byte is the lowest 8 bits. Decoded, each section is its name, a NUL byte, and a zlib stream.

The decoder

import os, re, sys, zlib

ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'

def decode(text):
 text = re.sub(r'[^A-Za-z0-9+/=]', '', text)
 out = bytearray()
 for i in range(0, len(text), 4):
 group = text[i:i + 4].rstrip('=')
 value = sum(ALPHABET.index(c) << (6 * j) for j, c in enumerate(group))
 out += value.to_bytes(3, 'little')[:len(group) * 6 // 8]
 return bytes(out)

src, outdir = sys.argv[1], sys.argv[2]
os.makedirs(outdir, exist_ok=True)
sections = re.findall(r'--BEGIN ROUTEROS SUPOUT SECTION\r?\n(.*?)--END ROUTEROS SUPOUT SECTION',
 open(src, encoding='ascii').read(), re.S)
for n, section in enumerate(sections):
 name, _, data = decode(section).partition(b'\0')
 text = zlib.decompressobj().decompress(data)
 safe = re.sub(r'[^\w.-]', '_', name.decode(errors='replace'))
 with open(os.path.join(outdir, f'{n:02d}_{safe}.txt'), 'wb') as f:
 f.write(text)
 print(f'{n:02d} {safe} {len(text)}')

python3 supout.py autosupout.rif supout/

The 6.49.5 file had 76 sections and the 7.23.8 one 87. The useful ones:

Section	What’s in it
`oops`	the kernel log of the last few crashes, each with its timestamp
`log`	the RouterOS log since boot
`resource`	`/system resource` plus `/proc/meminfo`
`.proc`	binary-ish dump of `/proc`, including `slabinfo`, `vmallocinfo`, `buddyinfo` (use `LC_ALL=C grep -a`)
`export`	the full configuration
`.debug`	per-service debug output

What it showed

On 7.23.8, every crash in oops ended the same way, after a process table in which every process had the same score:

[ 691.378048][ T142] [ 150] 0 150 1667 153 28672 0 -1000 parser
...
[ 691.378492][ T142] Out of memory and no killable processes...
[ 691.378509][ T142] Kernel panic - not syncing: System is deadlocked on memory

RouterOS runs every process with oom_score_adj -1000, so the OOM killer has nothing to kill and the kernel panics. Any out-of-memory, however brief, is a reboot. 6.49.5 got there by another route, with the kernel set to panic on any OOM: Kernel panic - not syncing: Out of memory: compulsory panic_on_oom is enabled.

The processes weren’t the problem, at about 0.5 MiB each. resource, written a minute after boot, was:

MemTotal: 118248 kB
MemFree: 7368 kB
SUnreclaim: 52756 kB

One minute into the boot, the kernel already held 52 MiB of unreclaimable memory. The caches in slabinfo add up to only about 10 MiB, so the rest is allocated outside them. In two of the panics, other CPUs were inside the Wi-Fi driver (ath_poll, __w_qos_decrypt_mpdu in the wlan and ath modules), which makes it the main suspect, but turning one radio off didn’t settle it.

The same reading works on any embedded Linux box: check oom_score_adj in the OOM dump, then SUnreclaim against what the process table and slabinfo explain.

Read the timestamps with care

The router has no battery-backed clock. After a crash, it restores an old time, and NTP takes minutes to catch up, so log and supout timestamps right after a reboot were about 7 minutes early. Five “crashes” stamped within two seconds of each other were most likely one boot loop.

The supout itself was written a minute after the crash. The e-mail service tried to send the crash notice before PPPoE and DNS were up, died, and that service failure is what produced the file (.debug shows error: DNS resolve failed [code: 21] and then died).

Before you send it to anyone

The decoded sections held, in plain text, the PPPoE password, the WireGuard private key, a TSIG key, and a bearer token from a dynamic DNS script. Rotate those before you hand the file to MikroTik support or attach it to a forum post.

On a router this tight, delete the file after you copy it. The root of the file system is in RAM, and a 480 KiB supout is RAM the kernel needed:

/file remove [find name~"autosupout"]

Takeaway

A supout.rif holds the kernel log, /proc and your config, and you can read it in a minute. Next time your MikroTik writes one, decode it and read oops before you open a support ticket.

Dynamic DNS for home routers the hacker way

Thu, 28 Oct 2021 17:48:15 +0200

Skip public DDNS services: Python can handle the same job. This post shows how to modify a DNS record in a zone file, upload the zone over SSH, and expose the update as a small web service.

The setup

A MikroTik router with a dynamic public IPv4 address
A name for the router in my own zone, on my primary DNS server
Python with dnspython, paramiko, Flask, and flask_jwt_extended

I first looked at dynamic DNS over a standard protocol, but I had to abandon the idea. MikroTik also offers its cloud service for a similar result, but I find it less convenient than a name in my own zone. That led me to a way to modify the zone file with Python.

The examples cover IPv4 only. Extending them to IPv6 is a small change.

Modify the zone file

The update has three steps: open the zone file, modify the record, and save the file.

def update(zone, name, newip4):
 original = f'zones/{zone}.zone'
 updated = update_dynamic_ip4(original, name, newip4, 3600)
 with open(original, 'w') as file:
 file.write(f'$ORIGIN {zone}.\n')
 file.write(updated)

The record update itself:

Load the zone file.
Find the record.
Replace it, or stop if the address hasn’t changed.
Update the SOA serial.
Return the zone as text.

Don’t forget the SOA serial. Without the bump, the secondary servers of your zone don’t see the change.

dnspython does the job, but its API is not my favorite cup of tea:

from datetime import date

import dns.rdataclass
import dns.rdataset
import dns.rdatatype
import dns.zone


class NoUpdate(ValueError):
 pass


def update_dynamic_ip4(zonefile, name, newip4, ttl):
 z = dns.zone.from_file(zonefile)
 new_data = dns.rdataset.from_text(dns.rdataclass.IN, dns.rdatatype.A, ttl, newip4)
 node = z.find_node(name)
 oldip4 = node.get_rdataset(dns.rdataclass.IN, dns.rdatatype.A).to_text().split()[3]
 if str(oldip4) == newip4:
 raise NoUpdate
 node.replace_rdataset(new_data)

 soa = z.find_node("@").find_rdataset(dns.rdataclass.IN, dns.rdatatype.SOA)
 soa_updated = update_soa(soa)
 z.find_node("@").replace_rdataset(soa_updated)
 return z.to_text(relativize=False)

I keep the serial as a date (%Y%m%d) and a two-digit revision. If the serial is from today, the revision increases by one; otherwise, it starts at zero for today. That allows 100 changes a day, plenty for one dynamic address.

def update_soa(soa):
 soa_parts = list(soa.to_text().split())
 version = soa_parts[-5]
 today = date.today().strftime("%Y%m%d")
 increment = 0
 if version.startswith(today):
 increment = int(version[len(today):]) + 1
 version = f'{today}{increment:02d}'
 soa_parts[-5] = version
 soa_new = " ".join(soa_parts[3:])
 ttl = soa_parts[0]
 return dns.rdataset.from_text(dns.rdataclass.IN, dns.rdatatype.SOA, ttl, soa_new)

Upload the zone file and reload the DNS server

By hand, you’d SSH to the DNS server, replace the file, and reload the server. paramiko does the same:

import paramiko


def upload_zone(localfile, remotefile, remotehost, remoteport, keyfile):
 with paramiko.SSHClient() as client:
 client.load_system_host_keys()
 client.connect(remotehost, remoteport, key_filename=keyfile)
 with client.open_sftp() as ftp:
 ftp.put(localfile, remotefile)
 client.exec_command('nsd-reload') # the reload command of your DNS server

The DNS server’s host key must already be in your known_hosts.

Expose it as a REST API

The router sends a request, and the app updates the zone. The new address comes from the request itself, and the router identifies itself with a JWT. remotefile, remotehost, remoteport, and keyfile come from the configuration.

This example creates a new secret and token each time the app starts, so the router needs the new token after a restart:

import uuid

from flask import Flask, request
from flask_jwt_extended import JWTManager, create_access_token, get_jwt_identity, jwt_required

app = Flask(__name__)
app.config["JWT_SECRET_KEY"] = str(uuid.uuid4())
jwt = JWTManager(app)
with app.app_context():
 token = create_access_token(identity={'name': name, 'zone': zone},
 expires_delta=False)

The endpoint:

@app.route('/api/update/ipv4', methods=['POST'])
@jwt_required()
def update_endpoint():
 identity = get_jwt_identity()
 name = identity['name']
 zone = identity['zone']
 if 'X-Forwarded-For' in request.headers:
 newip = request.headers['X-Forwarded-For']
 else:
 newip = request.remote_addr
 try:
 update(zone, name, newip)
 except NoUpdate:
 return {'status': 'unchanged'}
 upload_zone(f'zones/{zone}.zone', remotefile, remotehost, remoteport, keyfile)
 return {'status': 'updated'}

Most periodic calls find the address unchanged, so NoUpdate is the common path, not an error. Read X-Forwarded-For only when the app runs behind your own reverse proxy that sets it; otherwise, any client can put an arbitrary address there. Before deploying, add input validation and handling for the remaining exceptions.

Once deployed, set up the router to call the endpoint periodically.