My adventures erasing an Apple Mac equipped with a T2 security chip
Part 1. The adventure begins!
This story begins when I needed to prepare my Macbook Pro (2018, 13" with Touchbar) to go in for repairs, to have the keyboard replaced due to the well-known repeating/non-responsive keys issue. I started by making a TimeMachine backup of the laptop, in preparation for doing a fresh install of macOS. I wasn't about to send in my laptop with any of my personal information on it after all. This is where my problems began...
I had downloaded the latest version of macOS Big Sur, and prepared a bootable USB drive containing the installer as per Apple's instructions. I intended to boot from it, however, I ended up opting to use the built-in macOS Recovery System (Cmd + R). I opened up Disk Utility, and formatted/wiped the entire drive in the Macbook. As it turns out, this is a bad idea to do on a T2-equipped Mac! This is because, as I understand it, the encryption keys for the T2 are stored 'somewhere' on the drive. When you format the drive, you end up also wiping out the Recovery Partition, and this sensitive information, thus requiring an external reinstall.
When you reboot after having wiped the drive, you're presented with a globe icon, and the option to connect to a Wifi network. This is Apple's Internet recovery system. I then attempted to boot up from the USB drive I'd created (by holding down Alt (Option) on startup), however, I had no luck with this. After a bit of research I realised that the likely cause was that I had neglected to enable the USB booting option in the Secure Startup Utility, via Recovery Mode, prior to wiping the system.
There seemed to be no way of enabling it without Recovery Mode, which I no longer had access to. At this point I resigned myself to using the Internet Recovery option, which, in theory, will download a basic recovery mode system, and then allow the installation of macOS from the Internet. Of course, this would be a lot slower than installing it via a USB drive.
Unfortunately, Internet Recovery seemed to be very intermittent and the error's it gave (e.g. -2101f, etc.) didn't help very much. I ended up trying a few things, such as removing the device from the "Find My" app on my phone, and from my account via the iCloud website (even though I had disabled "Find My", and signed out of iCloud before starting this process!). I tried it via ethernet connection, and Wifi. Eventually, after many attempts and much patience, it succeeded and allowed me to install a new copy of macOS. I did this, setup the guest account for Apple, and sent the laptop in for repairs...
Part 2. The adventure continues...
Four weeks later, I finally got the laptop back (no thanks to UPS!), and that's where the real fun began... and what prompted me to write this blog post.
For security reasons, I wasn't about to trust a device that had been in someone else's control, even Apple's and/or a repair shop's. For this reason, I decided to first test it out, the keyboard had been replaced and it worked just fine. I then planned to wipe the drive again, and do a fresh install of macOS. This time, I thought I'd learnt my lesson, and I made sure to boot into Recovery Mode first, and enable booting from USB devices via the secure startup utility. I then proceeded to wipe the drive from Disk Utility in Recovery Mode again, and hope for the best...
Once again, I was presented with the Internet recovery globe upon reboot. Well, "that's strange", I though. I then proceeded to try my USB drive in every USB port, and.. nothing! I still only received the Internet Recovery startup screen. At this point I thought there might be something wrong with my USB or the installer (I had created it from my M1 Mac Mini, and this was an Intel Macbook). So, I downloaded Big Sur, and created a new bootable USB from my old Intel based Macbook.
This didn't work either! I tested both the USB drives in my other systems, and they were detected and presented as bootable options just fine. I'm still not sure exactly why this happened, my only guess is that I should have also lowered the Secure Boot security level in the security startup settings screen where I enabled USB booting. My advice here is to beware of this if you have a T2-based Mac device. Double check that you can boot from the USB drive you created, before doing anything drastic, like formatting your main drive!
At this stage I'd decided that I wasn't going to get USB booting working, and resolved to relying on the Internet Recovery option. This had worked before, the first time around, albeit after many attempts. I tried it multiple times over a few days, using Wifi and Ethernet (with different adaptors and switch ports), and each time it failed after between 30 minutes and 2 hours. It always failed with the -2101f error. Which, if you look that up online, doesn't lead you to anything too useful. Advice saying to create a bootable USB, which obviously wasn't going to work in my case.
I did some research, and found and followed a few of the standard suggestions, such as resetting the NVRAM/PRAM, resetting the SMC, etc.. but none of these seemed to work, nor behave like they were supposed to. I even went as far as following Apple's instructions to revive or recover an Apple device using a USB-C cable attached to another Mac device, via the Apple Configurator software. I followed the instructions, and did a revive on the Macbook. This was an interesting experience, and something that I didn't realise was possible! However, it didn't seem to help at all in my case.
I did eventually find a few forum threads, and one blogpost, with people having similar issues. The blog post I found noted that all -XXXX error numbers from Internet Recovery indicate some kind of network issue. They suggested trying Wifi, as Mac Ethernet adaptors can be unreliable, and if all else fails, to try it on a completely separate network, such as the hotspot from your iPhone. I setup my mobile hotspot, and connected my Macbook to it the next time I booted up. This time, the progress bar on the Internet recovery screen actually moved! Albeit, ever so slowly, and the 'estimated time remaining' number also changed this time around. However, after a few hours of very little progress, I gave up on this option.
I tried to think what else it could be... perhaps something on my ISP's side, or to do with my network, that was blocking Internet Recovery from working? This is when I realised that in between doing this for the first time, I had replaced my Ubiquiti USG with a Protectli Vault running pfsense! I hadn't customised much in pfsense yet, besides for adding some recommended extensions, so I thought it unlikely to be the problem, but this was my only lead. I decided to disable both pfBlockerNG-Devel, and Snort, on my firewall and try again. Low and behold, this time, via Ethernet, the progress bar moved appreciably and the time remaining also started to move. This eventually failed, but with a different error this time, -5101F.
I decided to try once more, using Wifi this time, and it worked just fine and booting up into Recovery Mode! I left it installing Big Sur overnight, and woke up to a fresh install! :-)
So, after many wasted hours, and some help and much encouragement from @Em0nTw1tter and @HypnInfosec, I decided to make the most of things, and to write up a blog post with what I'd learnt. Hopefully, this post will help someone else, and save them from many wasted hours of frustration and confusion!
Takeaways from my experience:
- Formatting the internal drive is a lot more dangerous to do on a T2-equipped Mac than on an older Mac, or a Windows or Linux PC.
- If you plan to boot and install macOS via a USB, first enable booting to USB from Recovery Mode, lower the Secure Boot security level, and make sure that you TEST THAT IT WORKS before proceeding.
- If you run into Internet Recovery errors (specifically high negative numbers), make sure your network is allowing the traffic through. Try using Ethernet and Wifi.
- Try using another network, if you can, to determine if the problem is with your network, or somewhere else.
- As always, make sure you have a Time Machine (or other) backup before starting.
- Be sure to leave yourself enough time, Internet recovery takes hours, even when it works the first time.
If you made it this far, thanks for reading! :-) I hope you found this post useful.