I made a few small bug fixes and released an update today, but now the Compatibility has randomly changed to iPhone 5s or later and I have no idea why. Previously it worked on any iOS 8.0+ device, and the changes I made were very small bug fixes.
After looking at the compatibility reference (https://developer.apple.com/library/ios/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/DeviceCompatibilityMatrix/DeviceCompatibilityMatrix.html ) there are only two things that are iPhone 5s+, and that's 'metal' and 'opengles-3'. I don't know what these are and how I could have accidentally used them? In my info.plist only "armv7" is in "Required device capabilities".
EDIT: It seems that the app now only supports devices with x64 processors (iPhone 5S onward and iPad Air onward). However, in xcode it says it still supports armv7...
So I found out you can check which architectures are included in a binary by running
xcrun lipo -info /App.app/app
Which told me only arm64 was included in this particular binary for whatever reason... I re-archived the app and it now has arm7 as well.