Race day dressing is its own very particular sport and some people take it extremely seriously. Royal Ascot has actual rules. Cheltenham is slightly more relaxed but still very much an occasion. Even a standard Saturday at your local track has an unspoken agreement that you are going to make an effort. The classic combination of a fitted dress and a fascinator exists for good reason because it works reliably well. What you want to avoid is anything too casual, anything that will be destroyed by the weather, and anything that photographs badly because race days are extremely well documented occasions whether you like it or not.