It seems like Brunei gets a bad rep on reddit for being boring, but honestly I had a great time here!
Beautiful islamic style architecture; dense rainforests with an enormous canopy walk and unusual looking creatures like the proboscis monkey; the world's largest (and ricketiest) water village at Kampung Ayer; and night markets with some genuinely unique and delicious food (like Ambuyat, made from the trunk of a palm tree).
I think 3 days here is ideal. That'd be enough time to comfortably:
Could do 4-5 max if you have niche interests like checking out the Seria oil fields in KB (I have a background in petrochemicals, so seeing the flares, rigs, and nodding donkeys right next to where people live was super interesting to me), but I think that's more than enough time.
Note that the country is Islamic, so bars/clubbing is off the cards. (However, the local Chinese-Bruneian community seems to have lots of open house parties, so if you know any locals you might get invited to one of those.) With that being said, it's very safe in terms of crime, and female travellers don't need to wear headscarfs or anything like that except maybe in the mosques.
Overall, I'd say it's perfect when combined with a larger trip to Borneo, like Sabah to see the Orang-utans and do some diving. Any questions, feel free to ask!