kernel_task kernel panic with Mountain Lion under pressure

This forum is to find answers to problems you may be having with ZEVO Community Edition.

Moderators: jhartley, MSR734, nola

kernel_task kernel panic with Mountain Lion under pressure

Post by grahamperrin » Sun Sep 23, 2012 10:55 pm

MacBookPro5,2 with 8 GB memory, OS X 10.8.2 and ZEVO Community Edition 1.1. Kicking the system much more roughly and extensively than I normally would, with the intention of a kernel panic.

Panic occurred after kicking for more than four hours. Harshness by me included combinations such as:

  • two concurrent backups of the CoreStorage encrypted JHFS+ startup volume (Time Machine to ZFS with copies=2 and compression=on on an imperfect device on FireWire, coinciding with SuperDuper! to a CoreStorage encrypted JHFS+ part of a more trusted device on the same bus)
  • MacPorts building apple-gcc42 and more to that startup volume during the backups
  • Windows 7 with 1 GB memory in VirtualBox writing as much and as often as possible to ZFS with compression=on on USB
  • QuickTime Player with a relatively large movie (probably ~225 MB read from osxfusefs)
  • Activity Monitor, Camouflage, Console, HardwareGrowler, Notes, Preview, TextExpander, TextWrangler, Wuala, System Preferences and a variety of other stuff
  • Safari, which might have become sluggish under all of the above, especially when application switching or creating new tabs and windows; but the app behaved perfectly.

Flip side, the good news is that 10.8.2 with this edition of ZEVO performed exceedingly well before the panic – much more smoothly than with any previous combination.

Code: Select all
panic(cpu 0 caller 0xffffff8005ab7bd5): Kernel trap at 0xffffff7f85fa8279, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x000000000000001d, CR3: 0x00000001441a1000, CR4: 0x0000000000000660
RAX: 0x000000000000001d, RBX: 0x000000000000001d, RCX: 0x0000000000000000, RDX: 0x0000000000000000
RSP: 0xffffff811d27bd20, RBP: 0xffffff811d27bd50, RSI: 0xffffff811d27be50, RDI: 0xffffff801bceaa30
R8:  0x0000000000000000, R9:  0x0000000000000000, R10: 0x0000000000000002, R11: 0x00007fff5fbe4770
R12: 0xffffff8019cf4000, R13: 0xffffff801bceaa30, R14: 0xffffff811d27be50, R15: 0xffffff801bceaa30
RFL: 0x0000000000010206, RIP: 0xffffff7f85fa8279, CS:  0x0000000000000008, SS:  0x0000000000000000
Fault CR2: 0x000000000000001d, Error code: 0x0000000000000000, Fault CPU: 0x0

Backtrace (CPU 0), Frame : Return Address
0xffffff811d27b9c0 : 0xffffff8005a1d626
0xffffff811d27ba30 : 0xffffff8005ab7bd5
0xffffff811d27bc00 : 0xffffff8005ace4ed
0xffffff811d27bc20 : 0xffffff7f85fa8279
0xffffff811d27bd50 : 0xffffff7f8603f9d9
0xffffff811d27be80 : 0xffffff7f86036e38
0xffffff811d27bec0 : 0xffffff8005b120af
0xffffff811d27bef0 : 0xffffff8005d40220
0xffffff811d27bfb0 : 0xffffff8005ab26b7
      Kernel Extensions in backtrace:
         com.getgreenbytes.filesystem.zfs(2012.9.14)[8241F82D-269B-30AD-86AB-AB7C3D520F2B]@0xffffff7f85fa5000->0xffffff7f86141fff
            dependency: com.apple.iokit.IOStorageFamily(1.8)[5BA4CD36-E96D-3A9E-ADFF-A863BBD63BC7]@0xffffff7f85f78000

BSD process name corresponding to current thread: kernel_task
Boot args: -v

Mac OS version:
12C54

Kernel version:
Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64
Kernel UUID: 69A5853F-375A-3EF4-9247-478FD0247333
Kernel slide:     0x0000000005800000
Kernel text base: 0xffffff8005a00000
System model name: MacBookPro5,2 (Mac-F2268EC8)

System uptime in nanoseconds: 27106418604491
last loaded kext at 334088512566: com.apple.filesystems.smbfs   1.8 (addr 0xffffff7f87fed000, size 229376)
last unloaded kext at 267799675487: com.apple.driver.AppleFileSystemDriver   3.0.1 (addr 0xffffff7f87da4000, size 8192)


Over to Wuala for the .panic file in full (plus sixty-something screenshots, which I don't expect people to browse; I began that series of shots whilst considering other users' reports of slow reads and writes).

Thoughts

Suspect that without active use of VirtualBox (4.2.0 at this time), there would have been no kernel panic. But I'm not finger-pointing, that's just a hunch.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

additional observations

Post by grahamperrin » Tue Sep 25, 2012 2:06 am

A late addition to the collection of things on Wuala: post-panic observation of malformation of the database disk image for MacPort's use of sqlite.

From a Terminal window restored by the OS, it's almost certain that MacPorts was building php5 at the moment of panic.

I set aside a copy of the 85.7 MB registry.db in case it's required in connection with this panic report.

(I worked around the malformation by restoring from a backup made by Time Machine.)
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

(2012.09.23) review, a question for Don or developers

Post by grahamperrin » Fri Sep 28, 2012 1:22 am

Don, please: should I attempt to reproduce this panic with 1.1.1?

Or are the diagnostics to help chase down a rare panic in zio stack not for cases such as this?

If you wonder about the level of hard disk imperfection mentioned in the opening post: my devices with known imperfections or suspected issues.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

2012-10-03 21:20 kernel_task kernel panic and cross referenc

Post by grahamperrin » Wed Oct 03, 2012 8:26 pm

grahamperrin wrote:… attempt to reproduce this panic with 1.1.1 …


The same KEXT and dependency in backtrace (2012.9.14 build of com.getgreenbytes.filesystem.zfs in the 2012-09-23 panic above, 2012.9.23 build in the 2012-10-03 panic below):

Code: Select all
panic(cpu 0 caller 0xffffff8001eb7bd5): Kernel trap at 0xffffff7f823a7b8d, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x00000000baddecaf, CR3: 0x000000021e2ad000, CR4: 0x0000000000002660
RAX: 0x00000000baddecaf, RBX: 0x00000000baddecaf, RCX: 0x0000000000000000, RDX: 0x0000000000000000
RSP: 0xffffff8114773d20, RBP: 0xffffff8114773d50, RSI: 0xffffff8114773e50, RDI: 0xffffff8029f12a30
R8:  0x0000000000000000, R9:  0x0000000000000000, R10: 0x00000000ffffffff, R11: 0x00000000ffffff80
R12: 0xffffff8014ca0000, R13: 0xffffff8029f12a30, R14: 0xffffff8114773e50, R15: 0xffffff8029f12a30
RFL: 0x0000000000010206, RIP: 0xffffff7f823a7b8d, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x00000000baddecaf, Error code: 0x0000000000000000, Fault CPU: 0x0

Backtrace (CPU 0), Frame : Return Address
0xffffff81147739c0 : 0xffffff8001e1d626
0xffffff8114773a30 : 0xffffff8001eb7bd5
0xffffff8114773c00 : 0xffffff8001ece4ed
0xffffff8114773c20 : 0xffffff7f823a7b8d
0xffffff8114773d50 : 0xffffff7f8243f311
0xffffff8114773e80 : 0xffffff7f82436770
0xffffff8114773ec0 : 0xffffff8001f120af
0xffffff8114773ef0 : 0xffffff8002140220
0xffffff8114773fb0 : 0xffffff8001eb26b7
      Kernel Extensions in backtrace:
         com.getgreenbytes.filesystem.zfs(2012.9.23)[04497DBB-8849-31D8-8496-BE10E5711C53]@0xffffff7f823a5000->0xffffff7f8253ffff
            dependency: com.apple.iokit.IOStorageFamily(1.8)[5BA4CD36-E96D-3A9E-ADFF-A863BBD63BC7]@0xffffff7f82378000

BSD process name corresponding to current thread: kernel_task
Boot args: -s -v

Mac OS version:
12C54

Kernel version:
Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64
Kernel UUID: 69A5853F-375A-3EF4-9247-478FD0247333
Kernel slide:     0x0000000001c00000
Kernel text base: 0xffffff8001e00000
System model name: MacBookPro5,2 (Mac-F2268EC8)

System uptime in nanoseconds: 29480214617277
last loaded kext at 23950646108620: com.apple.iokit.IOSCSIBlockCommandsDevice   3.5.1 (addr 0xffffff7f84402000, size 86016)
last unloaded kext at 24129641278687: com.apple.driver.AppleUSBCDC   4.1.22 (addr 0xffffff7f843ee000, size 12288)


http://www.wuala.com/grahamperrin/publi ... de=gallery for:

  • the 2012-10-03 .panic file in full
  • other files
  • links to two other incidents that I now treat as comparable.

Before the panic:

  1. Ubuntu in VirtualBox 4.2.0, I allowed a round of updates
  2. iTunes full screen presenting an episode of Horizon
  3. I fell asleep whilst watching iTunes, probably after Ubuntu had finished its updates.

This VirtualBoxVM uses
/Volumes/zhandy/Users/gjp22/Library/VirtualBox/Machines/Ubuntu/Ubuntu-11.10.vdi

zhandy is the sole pool and primary dataset on a StoreJet Transcend on USB, frontmost port of the MacBookPro5,2.

Other connections at the time of the panic, probably negligible: MagSafe and analogue audio out.

/Volumes/zhandy/Users/gjp22/Library/VirtualBox/Machines/Ubuntu/Logs/VBox.log
modified at
21:18 and I reckon that panic occurred at or shortly after
21:20

Post-panic review

I awoke to find the laptop cold and apparently not waking, which made me suspect that a panic had occurred.

A single press on the power button, boot was not verbose – peculiar, because yesterday I had reset NVRAM/PRAM (a rarity) then certainly set
boot-args="-v"

/Volumes/swap
on JHFS+ had more than one swap file – another peculiarity.

/private/var/vm
was empty (as expected with my configuration).

I restarted in single user mode, cautiously ran AppleJack (never tried it before with Mountain Lion), responded to its prompt about swapfile location. Allowed it to check the startup volume 'OS' (apparently OK) and to clean caches, nothing more.

Around this time (a little sleepy) I set, again, boot-args="-v" in NVRAM and restarted the computer.

AFAIR it restarted itself automatically – not a panic, maybe a response to my clearance of caches (side note, not an issue with ZEVO: this computer behaves similarly, restarts itself repeatedly, whenever I attempt safe boot).

Console view of the last two lines logged before the panic:

Code: Select all
2012-10-03 21:20:09.973 com.growl.hardwaregrowler.GNTPClientService[48959]: <GrowlGNTPNotificationAttempt: 0x400126ac0> failed because Error Domain=GCDAsyncSocketErrorDomain Code=7 "Socket closed by remote peer" UserInfo=0x40012fd60 {NSLocalizedDescription=Socket closed by remote peer}
2012-10-03 21:20:09.975 HardwareGrowler[675]: Failed with code 0, "(null)"


Of greater interest to me, around that time:

Code: Select all
2012-10-03 21:18:33.000 kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
2012-10-03 21:20:04.000 kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
2012-10-03 21:20:05.000 kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.


Compare those three (16 is usually diskarbitrationd) with the final line before panic for
diskimages-helper kernel panic, image on ZEVO on FireWire
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

again: VirtualBoxVM, iTunes in front whilst I slept

Post by grahamperrin » Wed Oct 24, 2012 1:22 pm

Before the panic:

  1. Ubuntu in VirtualBox 4.2.0, I allowed a round of updates
  2. iTunes full screen presenting an episode of Horizon
  3. I fell asleep whilst watching iTunes …


Recently, another incident involving VirtualBox and iTunes.

On the night of Monday 2012-10-22 I ran a VirtualBoxVM (4.2.2) with Ubuntu 12.04 either upgrading, or preparing to upgrade to 12.10. iTunes full screen, I fell asleep.

When I awoke in the early hours of Tuesday 2012-10-23 I found the computer off, not asleep. No .panic file, I guess the OS failed to save it, but all other indications are that a kernel panic occurred at or shortly after 23:57 on the 22nd. Maybe after iTunes finished its full screen presentation.

(After the incident, my next use of iTunes with the same video began playback from the beginning; not from the last point watched.)

For reference only, http://www.wuala.com/grahamperrin/publi ... de=gallery includes a variety of folders and files. Not annotated, but if explanations are required I'll be happy to add comments to selected items.

As with previous incidents, the final lines in system.log draw my attention to process 16, probably diskarbitrationd:

Code: Select all
2012-10-22 23:56:03.000 kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
2012-10-22 23:56:03.000 kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
2012-10-22 23:56:50.000 kernel[0]: aio_queue_async_request(): too many in flight for proc: 16.
2012-10-22 23:57:17.000 kernel[0]: touchup IPv4 Multicast with DSCP <0x30>


Following the incident, I did my best to recreate the environment that was in use immediately before the incident. http://www.wuala.com/grahamperrin/publi ... de=gallery includes a Mission Control view of that recreation.

Postscript

Shots at http://www.wuala.com/grahamperrin/publi ... de=gallery show Ubuntu unusable in the VirtualBoxVM, so I assume that panic occurred before completion of the upgrade from 12.04 to 12.10. (I used a ZFS snapshot to regain a .vdi with a usable installation of Ubuntu.)

Attention is drawn to file modification date logic for the affected virtual disk image.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

VirtualBoxVM window behaviours

Post by grahamperrin » Wed Oct 24, 2012 2:24 pm

As a VirtualBoxVM features in two or more incidents – and as panic may occur after exit from full screen view of a different app – it may be useful to note a peculiarity with windowing behaviours of VirtualBox.

When one VirtualBoxVM runs normally, GrabFS finds two windows:

  • the GrabFS TIFF for one of the two is easily recognisable as the window of the VirtualBoxVM
  • the GrabFS TIFF for the other window can be neither viewed nor copied (example).

Reference: GrabFS: The Screenshot File System
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

cross reference

Post by grahamperrin » Fri Nov 02, 2012 1:02 am

Remarkably similar: WebProcess panic whilst VirtualBoxVM used a .vdi on ZEVO

Don and colleagues: are these .panic files alone useful?

Or for Mountain Lion, do you need kernel panics to be more easily reproducible? (I recall your observation that generic panics are impossible to decode due to 10.8 kernels no longer having static addresses.)

I'm not chasing. Simply curious. Thanks.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

cross reference

Post by grahamperrin » Mon Nov 26, 2012 10:25 pm

grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

normalisation=none

Post by grahamperrin » Thu Dec 06, 2012 1:20 pm

Maybe not relevant to the panics, but for the record: I realised that one of the file systems used by me, for some things relating to VirtualBox, was without formD normalisation.

(Created on 2012-03-14, probably by by ZEVO Setup Assistant a few days before it gained the improvement for formD.)
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Christmas Day review

Post by grahamperrin » Tue Dec 25, 2012 6:39 am

No recent panics.

A while ago I set aside two USB cables that looked fine, but began behaving erratically. As usual, I have cats to thank for leaning on connections at appropriate moments.

One of the two cables was to a StoreJet Transcend, this typically had a direct connection to the MacBookPro5,2.

The other cable was to a hub, Sitecom USB 2.0 Dock CN-022, where I have a Seagate GoFlex Desk.

For a while now I have used both the Seagate and the StoreJet on that old hub – without problems.
grahamperrin Offline

User avatar
 
Posts: 1596
Joined: Fri Sep 14, 2012 10:21 pm
Location: Brighton and Hove, United Kingdom

Next

Return to Troubleshooting

Who is online

Users browsing this forum: CharlesMege, zgsirfst and 1 guest

cron