In 1447 the town and the castle were plundered by Bohemian mercenaries. After that a moat appeared and the walls were reinforced. After several years, a local count Bernhard VII made the castle his residence. In 1549 his successor, Bernhard VIII, invited the architect who had built a number of Renaissance buildings in the Weser region. Detmold castle was rebuilt and became a bit like the Neuhaus Castle in Paderborn. In the German Wikipedia you can see how the castle looked like in 1633.
Detmold itself is a very nice town. There's an old town where you can find about 600 original and restored historical buildings. However, on Saturdays it is quite crowded because of the market. We found a beautiful street called the Alley with many nice houses that resemble embassy buildings. It doesn't belong to the tourist attractions, but looks amazing in autumn. The most beautiful houses are in the end the street.
If you walk along the canal, then here N 51 55.862 E 8 52.598 there is a park with a waterfall and a palace.
Not far from Detmold is a huge Hermann monument, the main attraction of the town. There's also a Skansen, open air museum.
Booklet about the town in English (PDF)
Getting there: We parked here N 51 55.779 E 8 52.465, then walked to the old town along a beautiful canal. Detmold has a train station, use bahn.de.