In terms of hardness, diamonds are very strong. It's because of their perfect cleavage that makes them easily susceptible to shattering from blunt force. So in terms of that, they are very brittle since they have many places to split apart at. Diamonds are not the only gems with perfect cleavage though so they aren't alone. Moonstone, Fluorite, Amazonite, and Topaz to name a few also have perfect cleavage and are also susceptible to shattering if struck with a sharp blow.
Whereas quartz type gems and rubies actually have a very low to no cleavage so they're more durable though less scratch resistant than diamonds but are still fairly hard (quartz is a 7 while diamond is a 10). Which makes sense since in the show they're solider type gems! Jades, Turquoises, and Opals also have no cleavage. And of course if the gem is not very hard, the scratches could get deep enough to cause breaks even if the natural cleavage is low.
So if you want to go by both hardness and toughness, I'd say Rubies are the best of both worlds - more durable to blunt force (no cleavage) and are not easily scratched (9 on the Moh's scale). Diamonds can't be scratched except by other diamonds but are weak to sudden impact while things like Jade are resistant to breakage but can get very scratched up easily.








