the working iron never rusts

NVDAGPUWakeHandler: Disabling the NVIDIA graphics card on a MacBook Pro 10,1 (2012 retina)

2018-09-09T18:05:00-05:00

On Friday my laptop stopped turning on normally, leaving me stuck with a black screen. The machine was still booting up (I could hear the chimes and the keyboard backlight worked), but no screen image or backlight whatsoever.

After taking it to the Apple Store, where they told me it was a Logic board failure, I learned that this is a known issue with this generation of MacBooks from 2011-2013. It stemms from the use of a lead-free solder that results in premature failure of the discrete graphics chips. Unfortunately, the replacement period ended in 2016.

However, laptops from this period also contain an integrated GPU as part of the Intel Core i5/i7 processor. This GPU is used during normal workloads to improve battery life, and the discrete NVIDIA GeForce 650M is powered down by a controller chip called the gmux. This made me to hope that by forcing the NVIDIA GPU to power down I could regain use of my laptop with only the integrated GPU. To do this, I adapted a macOS kernel extension derived from the source code for the Linux gmux driver. It’s available on Github as NVDAGPUWakeHandler.

Modifications to only use the Integrated GPU

Prefer Intel GPU on boot with NVRAM parameter

The MacBook boots with EFI, and hackers have found the following NVRAM parameter can be used to “prefer” the integrated GPU during startup:

sudo nvram fa4ce28d-b62f-4c99-9cc3-6815686e30f9:gpu-power-prefs=%01%00%00%00

For me, this restored the ability to boot up normally.

Disable NVIDIA GPU in BIOS

It is possible to power down the gpu during boot using a custom bootloader.

NOTE: The instructions for this need to be modified for the Retina MBP. The GRUB config for High Sierra / Mojave should be:

set timeout=10
menuentry "macOS" {
  outb 0x7c2 1
  outb 0x7d4 0x28
  outb 0x7c2 2
  outb 0x7d4 0x10
  outb 0x7c2 2
  outb 0x7d4 0x40

  outb 0x7c2 1
  outb 0x7d4 0x50
  outb 0x7c2 0
  outb 0x7d4 0x50
  exit
}

(Adapted from this post.)

After booting, I could see that only the integrated GPU was present in System Profiler:

Disable NVIDIA GPU after waking from sleep

The same sequence of signals can be executed from a Kernel Extension (kext) to keep the GPU disabled after waking from sleep. I forked AMDGPUWakeHandler and modified it to execute the appropriate sequence for the Retina MBP.

My fork is available on GitHub. To install the kernel extension, you’ll need to either sign it with your own Apple developer account or disable SIP by booting into Recovery mode (CMD+R during boot) and running:

$ csrutil disable

You can verify that SIP is disabled by running csrutil status.

Enable backlight after waking from sleep in High Sierra (10.13) / Mojave (10.14)

This is sufficient to get the Mac to boot; however, in High Sierra something has changed such that the screen backlight controller no longer turns on after waking from sleep. If you hold up a flashlight, you can see that the screen is on, but with no backlight it appears nearly black. You can also see that macOS no longer thinks there is any backlight control for this display:

Fortunately the Linux driver for the GMUX controller in the MBP contains code to support modifying the backlight setting. I adapted it for use in NVDAGPUWakeHandler to set the backlight to a usable level after waking from sleep. Note that it’s currently hard-coded in the Kext, so if you prefer a different brightness setting, you’ll have to adjust it as necessary.

Update: TankTheFrank on Github has contributed a patch that allows brightness to be controlled using the standard brightness keys! See NativeDisplayBrightness.

Conclusions

After installing the kernel extension, wake from sleep seems to be working correctly and I haven’t had any issues. The biggest insight for me was to see the amazing work that the Linux kernel devs have put in to reverse engineer and implement drivers for the rather unique MBP hardware. Despite having no experience with Linux kernel development the C code for apple-gmux.c was incredibly well-documented and easy to follow, making it easy to adapt and use in macOS.

References

[1] https://dhavaldalal.wordpress.com/2018/05/29/how-to-disable-failed-discrete-gpu-nvidia-geforce-gt-650m-for-macbook-pro-101-mid-2012-on-high-sierra-10-13-4/

[2] https://gist.github.com/blackgate/17ac402e35d2f7e0f1c9708db3dc7a44

[3] https://github.com/blackgate/AMDGPUWakeHandler

[4] https://forums.macrumors.com/threads/disable-a-failed-amd-gpu-on-a-2011-macbook-pro-grub-solution.2087527/page-7

[5] https://forums.macrumors.com/threads/force-2011-macbook-pro-8-2-with-failed-amd-gpu-to-always-use-intel-integrated-gpu-efi-variable-fix.2037591/page-81

[6] https://wiki.archlinux.org/index.php/MacBookPro10,x#Graphics_2

Docker for Mac Memory Leak

2018-09-08T18:05:00-05:00

It seems that the default settings in Docker for Mac do not enforce any log rotation for services running on the integrated Kubernetes. This led to a seeming memory leak that would eventually crash my computer for long-running services. While this is somewhat understandable, given that Docker for Mac is clearly targeted for development, it still seems like an unfortunate default.

You can enforce a max size for logs by editing $HOME/.docker/daemon.json to include:

{
  "log-opts": {
    "max-size": "10m"
  }
}

ZFS on macOS

2018-09-03T19:12:43-05:00

ZFS is a filesystem developed at Sun Microsystems for their Solaris operating system. This is an experience report from recently setting it up as the filesystem for my secondary storage drives on macOS.

In contrast to other common filesystems you may encounter, ZFS is fully transactional and copy-on-write (COW). At the most basic level, you can imagine it as if the filesystem were a database of small blocks, indexed in B-trees and checksummed for verification. Notably, the COW design allows nearly instantaneous “snapshots” to be recorded and incremental backups between them. It also includes the feature sets of Logical Volume Management (LVM), RAID (mdadm), caching (bcache/flashcache), and transparent compression out of the box.

At one point, ZFS seemed like the future of filesystems, with Apple intending to use it in OS X (see: Time Machine). Furthermore, Sun released ZFS as open source under the CDDL license, however due to some licensing controversy it was never merged into the mainline Linux kernel. Thus ZFS was squandered, never to be enjoyed as widely as might have otherwise been.

In recent years, it’s become possible to run ZFS on linux through the efforts of the ZFS on Linux project, and the incorporation of easy-to-install ZFS packages in the Ubuntu repos. A related project, OpenZFSOnOSX, brings ZFS to macOS by using a Solaris Porting Layer (SPL) to translate Solaris system calls for the Darwin kernel on macOS, allowing them to run ZFS code mostly unchanged. This obviously brings some performance penalty, but one that is hopefully not too significant on spinning disks, since they will likely be I/O bound by the disks anyway.

Installation

Installation was easy using the provided package on the OpenZFSOnOSX website, or

$ brew cask install openzfs

which installs a few kernel extensions:

$ kextstat | grep net.lundman
  1 0xffffff7f816aa000 0x498      0x498      net.lundman.kernel.dependencies.31 (12.5.0) CF3A4A39-BA8C-4DFF-8BA7-B3C04D69457E
  1 0xffffff7f816ab000 0x11f5000  0x11f5000  net.lundman.spl (1.7.2) 0AB91572-CACF-39DF-86B3-116FF8CDCB8E <69 7 5 4 3 1>
  1 0xffffff7f828b0000 0x2d2000   0x2d2000   net.lundman.zfs (1.7.2) 0F708776-FDC2-39C5-87CE-42CFF3C5DD48 <70 25 7 5 4 3 1>

Tooling

ZFS has some pretty amazing tooling in the zfs, zpool, and zdb commands.

zpool is used to create and manage pools of disks that logically present as a single volume (like LVM), whereas zfs is used to manage individual datasets, which are nested hierarchically within pools but may have different tunings (recordsize, etc).

One feature I found myself using a lot for monitoring was zpool iostat, which gives a current picture of I/O operations across the different media in the pool:

$ zpool iostat -v 3
                                                  capacity     operations     bandwidth 
pool                                            alloc   free   read  write   read  write
----------------------------------------------  -----  -----  -----  -----  -----  -----
pool0                                            408G  10.5T    399    202  8.76M  13.2M
  raidz1                                         408G  10.5T    399    202  8.76M  13.2M
    media-4381EF74-91A2-A841-9073-00DD77A83EAA      -      -    133     70  2.92M  4.39M
    media-A00A5F1E-98B4-AA4C-ABE6-D19D52451730      -      -    133     69  2.92M  4.39M
    media-3D6C5673-A976-544C-9206-31ED6F4986E4      -      -    133     62  2.92M  4.40M
logs                                                -      -      -      -      -      -
  media-F0A2BB1A-583A-4D31-A9FD-887C7B326862     260K  7.00G      0      0      5     11
cache                                               -      -      -      -      -      -
  media-599EF695-4A06-488E-8679-FA03ED260C8D     519M  92.3G      0      0      1  7.16K
----------------------------------------------  -----  -----  -----  -----  -----  -----

Configuration

After installation, I made a pool of my three drives using the zpool command:

$ zpool create pool0 raidz disk1 disk2 disk3

Striped, mirrored, or RAID-Z?

You may be familiar with the different RAID levels (RAID0, RAID1, etc). Striping in ZFS corresponds to RAID0, mirroring to RAID1, RAIDZ-1 to RAID5, and RAIDZ-2 to RAID6.

These different setups offer different tradeoffs in:

Performance (IOPS)
Performance (bandwidth)
Space efficiency
Resiliency to drive failure

For example, striping the disks in parallel offers the best performance, since all disks may be used independently in parallel. However, in this configuration the failure of any one disk renders then entire set unusable, since it will contain a 1/N of the data.

There are many in-depth looks at these different configurations, and even calculators, but it’s outside the scope of this article. Suffice to say, because it is my home computer and I am somewhat stingy / out of available drive bays, I chose RAIDZ-1 for my configuration of 3 x 4TB drives, resulting in ~8 TB of usable space while being able to tolerate failure of any one drive.

RAIDZ carries some notable performance limitations, however, these can be mitigated somewhat by using an SSD for both write-ahead logging and caching (i.e. by using the SSD for better low-latency random I/O). Since I have an SSD in my desktop as the main root device, I created a small 8GB partition to use for write-ahead logging (the so-called ZIL, or ZFS intent log; when split out onto an SSD sometimes referred to as an SLOG), and a medium-sized 100G partition to use as a cache. These were added to the pool with:

$ zpool add pool0 cache disk4s1
$ zpool add pool0 log disk4s2

Tuning for MySQL

My computer runs a somewhat large MySQL instance (~3TB data), and MySQL on ZFS is not a good match with the out-of-the-box default configuration. The main reasons for this are:

Mismatched page size: ZFS and MySQL both manage data in fixed-size pages, however, the default MySQL page size is 16K while the default ZFS page (called recordsize) is 128K. This leads to thrashing as updates to MySQL pages require read-modify-writes of the larger ZFS pages.
Redundancy: Both ZFS and MySQL perform write-ahead logging, checksumming, and optionally compression of their data, which is redundant in this case.

Percona has a great guide on how to tune ZFS+MySQL to play nicely together, and there are some additional notes on the OpenZFS Wiki. In the end, for my setup, I went with:

Two separate ZFS datasets mysql/data and mysql/logs.
For the data:
- recordsize=64k
- compression=lz4
- primarycache=metadata
- logbias=throughput
For the logs:
- recordsize=128k (the default)
- primarycache=metadata
MySQL tunings:
- innodb_page_size=64k
- skip-log-bin
- innodb_doublewrite=off
- innodb_checksum_algorithm=none

With these settings, we match the page size of MySQL to the recordsize of ZFS, and also disable some of the redundant consistency settings in MySQL (double-writing and checksumming) that are not strictly necessary since they are provided by ZFS.

In addition, because both MySQL and ZFS perform in-memory caching, I limited the size of ZFS’s in-memory (ARC) cache to 4GB and disabled data caching (primarycache=metadata) in favor of MySQL’s innodb_buffer_pool, which is reported to be 7-200% faster.

Memory leak?

The biggest issue I ran into was importing my large MySQL dataset. It seemed that ZFS was leaking memory (in Activity Monitor a ballooning kernel_task), even when I limited the size of the in memory ARC cache with kstat.zfs.darwin.tunable.zfs_arc_max. This would eventually degrade performance as both MySQL and ZFS would compete for available memory.

Thankfully, I eventually stumbled on a link to a Github issue with the following helpful configuration snippet:

sysctl -w kstat.zfs.darwin.tunable.zfs_arc_max=4294967296
sysctl -w kstat.zfs.darwin.tunable.zfs_arc_meta_limit=3221225472

sysctl -w kstat.zfs.darwin.tunable.zfs_arc_min=1610612736
sysctl -w kstat.zfs.darwin.tunable.zfs_arc_meta_min=1342177280
sysctl -w kstat.zfs.darwin.tunable.zfs_dirty_data_max=536870912

Apparently, when under intense write load (as when I was importing my MySQL dataset), the memory allocator in SPL can fail to release dirty pages promptly, leading to what appears to be a memory leak.

After applying these sysctls (and making them persistent by adding them to /etc/zfs/zsysctl.conf), the memory usage seems to be appropriately capped.

Conclusions

While ZFS on macOS is definitely still a bit edge, I found it reasonably reliable, with no observed kernel panics or other major issues. Performance definitely takes a hit, although the authors claim that it may improve in time: to date they have primarily focused on stability rather than performance (which seems wise given some of the backlash to early bugs in Btrfs). In addition, ZFS offers a RAID5-like configuration not otherwise available in macOS. I’ll be curious to see how it performs going forward, and whether or not any issues arise. I’m also interested in setting up a system to use ZFS snapshots to perform incremental backups offsite.

If you’re interested in diving into more technical details of ZFS, Chris Siebenmann has an excellent series of blogs about it.

‘Advanced Format’ Disks

2018-09-01T22:29:45-05:00

Last week, one of the drives in my desktop was having an unrecoverable read error, so I wanted to swap it out with another disk. I had a Samsung 4TB external USB drive lying around, so I popped it out of its case to put in the machine, and put the old drive (which still had some accessible data on it) in the USB enclosure. After plugging it into my laptop, the partitions appeared to be corrupted:

However, if I put the drive back in the desktop computer it seemed to be fine 🤔.

On the desktop, Disk Utility reports a 512-byte block size:

$ diskutil info /dev/disk1 | grep "Size:"
Disk Size:                4.0 TB (4000787030016 Bytes) (exactly 7814037168 512-Byte-Units)
Device Block Size:        512 Bytes

whereas it is clear that an empty file takes up 4K:

$ touch /Volumes/HD2/test.txt
$ stat -f "%k" /Volumes/HD2/test.txt
4096

Digging Deeper

A few years ago, as drives began to surpass 2TB, hard drives upgraded their sector size from the historical default of 512 bytes to Advanced Format, typically 4KB. It seems there are varying approaches to how this is exposed however.

Because many computers at the time assumed 512-byte sectors, some manufacturers exposed a 512-byte sector to controllers, even though the disk actually used 4K sectors. Some didn’t, which can lead to partition misalignment if the disk is moved to a different controller. Others have run into this problem too.

Cracking open a hex editor to look at the drive, it became apparent that the partition header for HFS was not aligned to one of the 4k blocks:

Source: @klanomath’s great answer on StackExchange

It seems like the drive was being addressed in the desktop as a 512-byte disk, whereas the USB controller only supports 4K. So when plugging the USB drive into my laptop, the OS searches for partitions but fails to find them because they are not aligned to 4K blocks.

Recovery

The simplest way to fix the partition map in this scenario is just to reformat the drive with Disk Utility, which will create a new partition table aligned to the appropriate block size.

After first making a full image of the drive in my desktop, I set out to see whether I could recover the partitions without needing to fully reformat and re-image the drive, which would take some time over USB. Thankfully, it was pretty easy to do with TestDisk, a utility for repairing partition tables.

TestDisk was able to find the partition header for the single HFS+ partition on the drive, and rewrite the partition table with the appropriate offsets. Thankfully, since the drive had only one partition, it was pretty easy to find and fix. I’ll probably keep the backup image around for a little while just in case.