Stress
If you don't have a love affair with numbers, the work can be painstakingly dull. In a small shop, there will be a lot of pressure on you to be on the ball and interpret the available data correctly.
As someone who's often a lone wolf at the company when it comes to job description, it really is all on you as the actuary. One minor error in formula can be the cause of millions in losses for the company you represent...which means hundreds of thousands in losses to you when they fire you.