Physical Danger
Thoroughbred racing is an extremely dangerous sport. Like we said—you're going really, really fast on the back of a really, really big animal and you have no emergency brakes or air bags of any kind.
Injuries are common, and at least two jockeys a year suffer spinal injuries from riding (source). That does put a damper on your future prospects, to say the least.
As if the dangers associated with riding the horse aren't enough, the pressure to stay under a certain weight can lead to eating disorders like bulimia or anorexia. This is so common the jockeys even have their own racing-related names for the diseases. But it's neither a healthy nor smart way to make a living, so if this is what it takes, you might want to consider a different career and do the horse-riding thing as an occasional hobby.