For the better part of the last year, I have been the principal engineer (and for most of the project, the *only* engineer) for an upcoming smart glasses product. I've had to customize the firmware, build the app for communicating with the glasses, and build the backend services. This is despite having no previous work experience in hardware, app development, or three of the six programming languages I would end up using (C++, Go, Python, Kotlin, TypeScript, Swift). I want to give credit where credit is due right at the outset: I could not have done this without the *massive* assistance of LLMs. The impact a single moderately skilled engineer can make in areas totally outside their realm of expertise is beyond what I thought possible. I was so surprised at the scope of what I have built so far that I decided to document it at a high level, primarily for my own later recollection: ## Dev Log We are sourcing glasses from China that run a simple RTOS (real-time operating system) and have dual waveguide displays, speakers, and microphones. The firmware is basic, primarily consisting of Bluetooth connection code, a GUI for navigating an app menu, and BLE commands to send content to the glasses. The compiled binary is about ~3mb. The apps that were in the firmware were basically templates. For example, there were pre-made UI components for chat bubbles, but anything that would appear in those chat bubbles had to be sent from the phone. The glasses themselves have no internet access. Thankfully the firmware edits were relatively minor, given the firmware is extremely painful to build and update OTA whenever I make a change. It's written in C++ and requires proprietary build software ($200 a month) to compile. I fixed some bugs (such as the pairing screen not being dismissible after the connection dropped temporarily), updated a couple UI components (through the extremely unintuitive "GUIX Studio"), and Americanized some aspects (display 12hr time formatting, temperature in Fahrenheit, etc.). The ability to dump the entire firmware into Cursor/Codex/Gemini CLI (whatever happens to be leading at the time) is priceless. It's a massive codebase whose structure I do not know, but I am able to ask things like "where are all the places that deal with navigation" and dig in from there. The app has been the real challenge. We decided to build it as an Expo app to be cross-platform across iOS and Android, but given the nature of the app, I have to dive into native code for each platform quite often. At the time of writing actually, I've really only done the Android side of things, with some cursory research on how I might approach it for iOS. If you're unfamiliar with how Bluetooth works, there's not just one protocol called "Bluetooth." There are a *ton* of protocols that a device could possibly use to communicate over Bluetooth. Discovering the intricacies of how to achieve consistent pairing, connecting, and communicating with the glasses was...a process. I had no documentation on the firmware besides an incomplete list of BLE (Bluetooth Low Energy) commands the glasses supported, and the rest I had to figure out by reading the firmware itself. And of course, this all had to be done through native code and is completely different on iOS and Android. For most of what I did, I was only concerned with BLE. This covered things like sending the text to display on the glasses or setting the brightness level. But I also had to make the glasses work as regular Bluetooth headphones. It took me a while to figure this out, but I eventually discovered the firmware blocked the ability to pair the glasses as headphones through a phone's native Bluetooth settings until the user had paired the glasses through the vendor's app, an intentional decision on the manufacturer's part. I knew that in order to work as headphones, I had to connect two other Bluetooth profiles: A2DP and HFP. However, they had made it so these profiles were not available until you had connected *another* profile called SPP through the app. This is different behavior than the Meta Ray-Bans for example, which you can pair as a regular Bluetooth headset without having to first pair the glasses through the Meta AI app. I think the theory was that customers would be confused if they paired the glasses through their phone's Bluetooth settings and audio worked, but then the glasses still complained about not being paired (since they still lacked the BLE connection). Anyway, even once I figured that out, managing pairing state was tricky because of what iOS and Android let you see about the status of Bluetooth devices connected to the user's phone. For example, Android lets you see which devices are "bound" to the device (have ever been paired) but doesn't let you see which are currently connected. iOS doesn't let you see anything. I had to create a flow in the app that would let the user establish BLE, SPP, A2DP and HFP connections to the glasses as simply as possible. On Android, this looked like sending a pairing request to the glasses, long-press confirming on the glasses (this completed BLE pairing), then adding the SPP connection, and then prompting the user to accept a pairing notification from the phone's native Bluetooth settings (for A2DP and HFP). As far as I can tell, this is as simple as I can make it, mirroring the behavior of the Meta Ray-Bans closely. And that was just the initial connection. Knowing the state of the glasses at all times (are they connected over BLE but not A2DP/HFP? are they disconnected from SPP? did they disconnect because the user unpaired them in their phone's Bluetooth settings, because they got too far from the phone, because they turned off Bluetooth, or...?) involved quite the state machine. But once I gave the phone the ability to talk to the glasses, I had to figure out what to say and how to say it. I had to figure out how to format the packets, how to sign them with CRC, how to split large payloads across chunks, what custom form of run-length encoding they were using for images, etc. etc. This was probably what I actually spent the most time on, because I actually worked on prototyping the app for a long time before I had access to the firmware itself. See [Sending Images to Smart Glasses via Bluetooth](Sending%20Images%20to%20Smart%20Glasses%20via%20Bluetooth.md) for a full description of how I tried reverse-engineering one of the commands. Once I knew how to communicate with the glasses, I needed to actually make them useful. We decided on an initial list of features we thought were achievable in a couple months: - Translation - Chat - Listener - Live Captions - Notifications - Calls - Weather - Music - Navigation I will probably just link to a separate post about how I built the backend services for Translation, Chat, Listener, and Live Captions. If there's not a link here yet, it's because I haven't written that post. The gist is it was originally written in Python, then rewritten in Go, and had some neat techniques to achieve translation across multiple languages simultaneously in real-time. The more standard smartwatch-like features that could live offline were more straightforward. Calling and Music were actually implemented entirely within the firmware already. But there were still a couple interesting elements: For example, on Android you can get permission to read the user's notifications from other apps, allowing you to forward to the glasses for display. So all you have to do on the app side of things is let the user select which notifications they want to see, and forward those. On iOS however, Apple does not allow you to read the notifications from other apps. They have their own internal API they use for this sort of functionality with the Apple Watch, but they do not allow third-party developers access to it. Instead, they have their own old, poorly-documented method of broadcasting these notifications over Bluetooth directly to the paired devices that require them, bypassing the app. This meant it was harder to give the user control over the types of notifications they wanted to receive on the glasses, because instead of just being a menu in the app, it had to be done via firmware on the glasses. Another interesting feature was navigation. The obvious way to approach this would be to use a navigation API like Google Maps and have some sort of navigation integration within the app. However, Google Maps is not free and I figured users would prefer to not have to switch how they navigated simply to get it to display in the glasses. And I did not want to have to implement a whole nav app if if I didn't have to. So on Android, I leveraged the ability to read notifications again, and simply extracted the directions data from Google Maps notifications and forwarded them to the glasses, where they would be parsed and displayed. On Apple, this was not possible as already stated, so I'm going to have to make an Apple Maps integration (thankfully the Apple MapsKit is free for use in iOS apps). Thus far I have MVPs of Translation, Chat, Listener, Notifications, and Navigation in the Android app, with Calling and Music being handled by the firmware. I have a teammate who might take implementing the iOS side of things, but they're busy on another project, so it's possible I'll be doing that as well. I've really enjoyed getting to stretch my abilities and will be thrilled if the glasses actually end up in the hands of consumers by the end of the year. *9.2.25*