u/dracoflar has made a pretty good thread for GPU recommendations, check it here: GPU Buyers Guide. While it's made for Catalina the information is applicable to both High Sierra and Mojave, the rest of this FAQ section is still relevant (with more sources).
BIG. FAT. NOTE: VGA and analog DVI on HD4000 and later, Nvidia or AMD GPUs IS NOT SUPPORTED. Use HDMI, DVI-D or DP/mDP.
Laugh all you want, the most compatible GPU of them all are intel GPUs!
So here is a list of the intel GPUs (only on Core CPU models):
- Intel GMA950: up to 10.7.5
- Intel X3100: up to 10.7.5
- Intel HD Graphics 1st generation (Arrandale): up to 10.13.x [NOT RECOMMENDED]
- Intel HD Graphics 2000: NOT SUPPORTED
- Intel HD Graphics 3000: up to 10.13.x
- Intel HD Graphics 2500 (IvyBridge): NOT SUPPORTED (only used for QuickSync)
- Intel HD Graphics 4000: up to Current
- Intel HD Graphics 4200-4400-4600-5000-5100-5200 (and Iris): up to Current
- Intel HD Graphics 5300-5500-5600-6000 (and Iris/Pro): up to Current
- Intel HD Graphics HD510 to 550: up to Current
- Intel HD Graphics P530 (and probably the P555 and P580): up to Current
- Intel HD Graphics HD610 (GT1-tier): NOT SUPPORTED
- Intel HD Graphics HD615 to HD650: up to Current
- Intel HD Graphics UHD620 to 655: up to Current
- Intel Iris Graphics IceLake: 10.15.4 to Current(Note that only Iris models like the G4 and G7 are supported, GT1 are not supported)
- ANY Intel HD Graphics in Atom/Celeron/Pentium: NOT SUPPORTED
- Tesla series: 8XXX and newer support High Sierra while 7XXX and older are not guaranteed past El Capitan
- Fermi series: Up to High Sierra though many experience graphical issues so are limited to Sierra
- Kepler: 10.8.3 to Current
- Maxwell: 10.10.0 to 10.13.6 (Needs WebDrivers)
- Pascal: 10.12.4 to 10.13.6 (Needs WebDrivers)
- Turing: Unsupported
TLDR: try to get an AMD card or use the intel GPU since the Nvidia is probably dying (Apple VS Nvidia hate is still ongoing).
- Navi 10: 10.15.1 to Current
- Vega 20: 10.14.5 to Current
- Vega 10: 10.13.3 to Current
- Polaris: 10.12.6 to Current
- Radeon R7/R9: 10.8.3 to Current
There are more supported cards but AMD's fragmented product stack makes it hard to organize. GPU Buyers Guide has a full list
Usually all motherboards are "compatible," as long as their chipset is Intel based (or AMD based in case of AMD). However there are some exceptions (if you're going to run macOS on a legacy system):
- VIA Motherboards (with VIA chipset, really old stuff)
- nForce Motherboards (with Nvidia nForce chipset): Mostly works but that's really old hardware
Also, one of the most important aspects of a properly supported motherboard is UEFI compatibility. This does not mean that older (Legacy) motherboards are unsupported, far from that, UEFI Motherboards will ease your life and most of them usually support newer hardware and probably a lot more compatible than older ones. Same with laptops.
Some motherboards with issues on macOS:
- 300 series Intel motherboards(excluding Z370):
No supported NVRAM requiring emulation, recently fixed
- Issues setting up the integrated GPU, requiring manual BusID patching
- Running AWAC clock which are unsupported in macOS requiring more patches
- MSI boards commonly having "Couldn't allocate runtime area"
- Guide for fixing this: Fixing KALSR slide values
- AsRock running non-native USB controllers
See Anti-Hackintosh Buyers Guide for more info.