hubertf's NetBSD Blog
Send interesting links to hubert at feyrer dot de!
 
[20080915] Debugging NetBSD kernel with qemu
Alexander Shishkin has written some posted instructions on how to debug a NetBSD kernel running in qemu with gdb. He reminds how to build NetBSD, provides a shell script for making a bootable disk, and explains the magic on how to run qemu and connect gdb to it. An example session shows how to load the kernel, set a breakpoint, and examine registers and variables when the breakpoint is reached.

[Tags: , , ]


[20061115] Post mortem debugging, or: what happened before it crashed? (Updated)
So your machine paniced, and as you were running X you have no clue what went on? Here's a nice way to find out, assuming you have a kernel crash dump. To ensure the latter, set kern.dump_on_panic=1 in /etc/sysctl.conf. Now, what to do with those crashdumps? 
% ls -l /var/crash/
total 3183838
-rw-r--r--  1 root  wheel          3 Nov  2 02:09 bounds
-rw-r--r--  1 root  wheel          5 Jun 30  2004 minfree
...
-rw-------  1 root  wheel  181265401 Nov  2 02:11 netbsd.26.core.gz
-rw-------  1 root  wheel    2162696 Nov  2 02:11 netbsd.26.gz
In /var/crash, "bounds" contains an increasing counter for the crashdump number (it would be "27" in the above example), and "minfree" contains the minimum amount of free space in kilobytes that should keep free - both files are read by savecore(8) when /etc/rc.conf has "savecore=yes", which is the default.

The actual crashdump consists of two gzipped files - the actual memory dump "netbsd.XX.core.gz" and a copy of the running kernel "netbsd.xx.gz". After uncompressing the files can be used for looking at the system at the point of it's panic: 

# gunzip netbsd.26*.gz
#
Note that the crashdump may contain sensitive data and is such only readable by root!

The crashdump can be read by programs that use libkvm to read through the crashdump's kernel memory, e.g. gdb(1), dmesg(8), ps(1), fstat(8), ipcs(1), netstat(8), nfsstat(8), pmap(1), w(1), pstat(8), vmstat(8) etc., using the -M and -N switches.

Some examples:

  • To show the system's message buffer at the time of the crash: 
    % dmesg -M netbsd.26.core -N netbsd.26
...
unmounting /home (/dev/wd1e)...
unmounting /tmp (mfs:371)...warning: mfs read during shutdown
dev = 0xff00, block = 10496, fs = /tmp
panic: blkfree: freeing free block
Begin traceback...
uvm_fault(0xcbfd07f0, 0x2000, 1) -> 0xe
fatal page fault in supervisor mode
trap type 6 code 0 eip c0305083 cs 8 eflags 10246 cr2 2900 ilevel 0
panic: trap
Faulted in mid-traceback; aborting...
dumping to dev 0,1 offset 2024327
dump 511 510 509 508 507 506 505 504 503 502 501 500 499 498 497 496
495 494 493 ...
    Apparently the system tried to free a block that was already fred here when umounting /tmp.

  • Display virtual memory parameters: 
    % vmstat -M netbsd.26.core -N netbsd.26 -s
     4096 bytes per page
        8 page colors
   127888 pages managed
          ...

  • Attach the GNU debugger gdb(1) to the system crash dumpQ, to poke around deeply: 
    % gdb netbsd.26
...
(gdb) target kcore netbsd.26.core
panic: blkfree: freeing free block
#0  0x0ac04000 in ?? ()
(gdb) bt
#0  0x0ac04000 in ?? ()
#1  0xc03084b5 in cpu_reboot ()
#2  0xc02a57aa in panic ()
#3  0xc0313127 in trap ()
#4  0xc0102dfd in calltrap ()
#5  0xc0182544 in db_get_value ()
#6  0xc03058f1 in db_stack_trace_print ()
#7  0xc02a577c in panic ()
#8  0xc0205db7 in ffs_blkfree ()
#9  0xc020b8d5 in ffs_indirtrunc ()
...
  • Unfortunately there are a number of programs that I didn't get to work with my crashdump, but that may be due to its point after/during system shutdown, e.g. ps(1) didn't work.
Still that should give some start for poking around...

Update: Apparently 'target kcore' was renamed to 'target kvm' in gdb6, see this posting.

[Tags: , , ]


[20060729] Enabling separate debug symbol support for gdb6
NetBSD-current got an update to the latest GNU debugger, gdb6, recently. That version allows keeping the debug information for a binary in a seperate file, thus not bloating every client's filesystem while still being able to keep debugging symbols around in some place.

Luke Mewburn has sent a proposal to enable this in NetBSD, and he has committed code now that allows setting "MKDEBUG=yes" to put seperate debugging symbols into /usr/libdata/debug.

This is currently not enabled by default, but maybe someone finds this useful, wants to test it and give some feedback.

[Tags: , ]


Tags: , 2bsd, 34c3, 3com, 501c3, 64bit, acl, acls, acm, acorn, acpi, acpitz, adobe, adsense, Advocacy, advocacy, advogato, aes, afs, aiglx, aio, airport, alereon, alex, alix, alpha, altq, am64t, amazon, amd64, anatomy, ansible, apache, apm, apple, arkeia, arla, arm, art, Article, Articles, ascii, asiabsdcon, aslr, asterisk, asus, atf, ath, atheros, atmel, audio, audiocodes, autoconf, avocent, avr32, aws, axigen, azure, backup, balloon, banners, basename, bash, bc, beaglebone, benchmark, bigip, bind, blackmouse, bldgblog, blog, blogs, blosxom, bluetooth, board, bonjour, books, boot, boot-z, bootprops, bozohttpd, bs2000, bsd, bsdca, bsdcan, bsdcertification, bsdcg, bsdforen, bsdfreak, bsdmac, bsdmagazine, bsdnexus, bsdnow, bsdstats, bsdtalk, bsdtracker, bug, build.sh, busybox, buttons, bzip, c-jump, c99, cafepress, calendar, callweaver, camera, can, candy, capabilities, card, carp, cars, cauldron, ccc, ccd, cd, cddl, cdrom, cdrtools, cebit, centrino, cephes, cert, certification, cfs, cgd, cgf, checkpointing, china, christos, cisco, cloud, clt, cobalt, coccinelle, codian, colossus, common-criteria, community, compat, compiz, compsci, concept04, config, console, contest, copyright, core, cortina, coverity, cpu, cradlepoint, cray, crosscompile, crunchgen, cryptography, csh, cu, cuneiform, curses, curtain, cuwin, cvs, cvs-digest, cvsup, cygwin, daemon, daemonforums, daimer, danger, darwin, data, date, dd, debian, debugging, dell, desktop, devd, devfs, devotionalia, df, dfd_keeper, dhcp, dhcpcd, dhcpd, dhs, diezeit, digest, digests, dilbert, dirhash, disklabel, distcc, dmesg, Docs, Documentation, donations, draco, dracopkg, dragonflybsd, dreamcast, dri, driver, drivers, drm, dsl, dst, dtrace, dvb, ec2, eclipse, eeepc, eeepca, ehci, ehsm, eifel, elf, em64t, Embedded, embedded, emips, emulate, encoding, envsys, eol, espresso, etcupdate, etherip, euca2ools, eucalyptus, eurobsdcon, eurosys, Events, exascale, ext3, f5, facebook, falken, fan, faq, fatbinary, features, fefe, ffs, filesystem, fileysstem, firefox, firewire, fireworks, flag, flash, flashsucks, flickr, flyer, fmslabs, force10, fortunes, fosdem, fpga, freebsd, freedarwin, freescale, freex, freshbsd, friendlyAam, friendlyarm, fritzbox, froscamp, fsck, fss, fstat, ftp, ftpd, fujitsu, fun, fundraising, funds, funny, fuse, fusion, g4u, g5, galaxy, games, gcc, gdb, gentoo, geode, getty, gimstix, git, gnome, google, google-soc, googlecomputeengine, gpio, gpl, gprs, gracetech, gre, groff, groupwise, growfs, grub, gumstix, guug, gzip, hackathon, hackbench, hal, hanoi, happabsd, hardware, Hardware, haze, hdaudio, heat, heimdal, hf6to4, hfblog, hfs, history, hosting, hotplug, hp, hp700, hpcarm, hpcsh, hpux, html, httpd, hubertf, hurd, i18n, i386, i386pkg, ia64, ian, ibm, ids, ieee, ifwatchd, igd, iij, image, images, imx233, imx7, information, init, initrd, install, intel, interix, internet2, interview, interviews, io, ioccc, iostat, ipbt, ipfilter, ipmi, ipplug, ipsec, ipv6, irbsd, irc, irix, iscsi, isdn, iso, isp, itojun, jail, jails, japanese, java, javascript, jetson, jibbed, jihbed, jobs, jokes, journaling, kame, kauth, kde, kerberos, kergis, kernel, keyboardcolemak, kirkwood, kitt, kmod, kolab, kvm, kylin, l10n, landisk, laptop, laptops, law, ld.so, ldap, lehmanns, lenovo, lfs, libc, license, licensing, linkedin, links, linksys, linux, linuxtag, live-cd, lkm, localtime, locate.updatedb, logfile, logging, logo, logos, lom, lte, lvm, m68k, macmini, macppc, macromedia, magicmouse, mahesha, mail, makefs, malo, mame, manpages, marvell, matlab, maus, max3232, mbr95, mbuf, mca, mdns, mediant, mediapack, meetbsd, mercedesbenz, mercurial, mesh, meshcube, mfs, mhonarc, microkernel, microsoft, midi, mini2440, miniroot, minix, mips, mirbsd, missile, mit, mixer, mobile-ip, modula3, modules, money, mouse, mp3, mpls, mprotect, mtftp, mult, multics, multilib, multimedia, music, mysql, named, nas, nasa, nat, ncode, ncq, ndis, nec, nemo, neo1973, netbook, netboot, netbsd, netbsd.se, nethack, nethence, netksb, netstat, netwalker, networking, neutrino, nforce, nfs, nis, npf, npwr, nroff, nslu2, nspluginwrapper, ntfs-3f, ntp, nullfs, numa, nvi, nvidia, nycbsdcon, office, ofppc, ohloh, olimex, olinuxino, olpc, onetbsd, openat, openbgpd, openblocks, openbsd, opencrypto, opendarwin, opengrok, openmoko, openoffice, openpam, openrisk, opensolaris, openssl, or1k, oracle, oreilly, oscon, osf1, osjb, paas, packages, pad, pae, pam, pan, panasonic, parallels, pascal, patch, patents, pax, paypal, pc532, pc98, pcc, pci, pdf, pegasos, penguin, performance, pexpect, pf, pfsync, pgx32, php, pie, pike, pinderkent, pkg_install, pkg_select, pkgin, pkglint, pkgmanager, pkgsrc, pkgsrc.se, pkgsrccon, pkgsrcCon, Platforms, plathome, pleiades, pocketsan, podcast, pofacs, politics, polls, polybsd, portability, posix, postinstall, power3, powernow, powerpc, powerpf, pppoe, precedence, preemption, prep, presentations, prezi, products, Products, proplib, protectdrive, proxy, ps, ps3, psp, psrset, pthread, ptp, ptyfs, Publications, puffs, puredarwin, pxe, qemu, qnx, qos, qt, quality-management, quine, quote, quotes, r-project, ra5370, radio, radiotap, raid, raidframe, rants, raptor, raq, raspberrypi, rc.d, readahead, realtime, record, refuse, reiserfs, Release, Releases, releases, releng, reports, resize, restore, ricoh, rijndael, rip, riscos, rng, roadmap, robopkg, robot, robots, roff, rootserver, rotfl, rox, rs323, rs6k, rss, ruby, rump, rzip, sa, safenet, san, sata, savin, sbsd, scampi, scheduler, scheduling, schmonz, sco, screen, script, sdf, sdtemp, secmodel, security, Security, sed, segvguard, seil, sendmail, serial, serveraptor, sfu, sge, sgi, sgimips, sh, sha2, shark, sharp, shisa, shutdown, sidekick, size, slackware, slashdot, slides, slit, smbus, smp, sockstat, soekris, softdep, softlayer, software, solaris, sony, sound, source, source-changes, spanish, sparc, sparc64, spider, spreadshirt, spz, squid, ssh, sshfs, ssp, statistics, stereostream, stickers, storage, stty, studybsd, subfile, sudbury, sudo, summit, sun, sun2, sun3, sunfire, sunpci, support, sus, suse, sushi, susv3, svn, swcrypto, symlinks, sysbench, sysctl, sysinst, sysjail, syslog, syspkg, systat, systrace, sysupdate, t-shirt, tabs, talks, tanenbaum, tape, tcp, tcp/ip, tcpdrop, tcpmux, tcsh, teamasa, tegra, teredo, termcap, terminfo, testdrive, testing, tetris, tex, TeXlive, thecus, theopengroup, thin-client, thinkgeek, thorpej, threads, time, time_t, timecounters, tip, tk1, tme, tmp, tmpfs, tnf, toaster, todo, toolchain, top, torvalds, toshiba, touchpanel, training, translation, tso, tty, ttyrec, tulip, tun, tuning, uboot, ucom, udf, ufs, ukfs, ums, unetbootin, unicos, unix, updating, upnp, uptime, usb, usenix, useradd, userconf, userfriendly, usermode, usl, utc, utf8, uucp, uvc, uvm, valgrind, vax, vcfe, vcr, veriexec, vesa, video, videos, virtex, virtualization, vm, vmware, vnd, vobb, voip, voltalinux, vpn, vpnc, vulab, w-zero3, wallpaper, wapbl, wargames, wasabi, webcam, webfwlog, wedges, wgt624v3, wiki, willcom, wimax, window, windows, winmodem, wireless, wizd, wlan, wordle, wpa, wscons, wstablet, X, x.org, x11, x2apic, xbox, xcast, xen, Xen, xfree, xfs, xgalaxy, xilinx, xkcd, xlockmore, xmms, xmp, xorg, xscale, youos, youtube, zaurus, zdump, zfs, zlib

'nuff. Grab the RSS-feed, index, or go back to my regular NetBSD page

Disclaimer: All opinion expressed here is purely my own. No responsibility is taken for anything.
Access count: 36035963
Copyright (c) Hubert Feyrer