It's not an area I've ever stopped in before, it has always just been a drive through but the harbour and seafront there are great and we would definitely stop there another time. To get to the fort you drive up quite a steep hill and the car parking is free (but be aware there is a 1.8 metre height restriction). At the top you can see out to sea and down to the harbour where there is a massive scrap metal yard and also enormous ferries come in to dock, the kids really liked it. We arrived about lunch time and had a picnic on the cliff top before going in as there are some very useful benches right by the parking.
Newhaven Fort is open from February to November 10am - 5pm each day with the last admission being at 4pm. I feel the entry cost was really reasonable at £19.95 for a family of 5 and if you book in advance (it must be at least the day before) you get 10% off the admission price.
We spent around four hours at the fort, some of this was because we choose to sit on the clifftop and relax in the sun as a family and also to have a coffee in the 1940's tea room. I would say it is probably a couple of hour attraction normally for a family that likes to have a look around but not to read every piece of history (you could be there all day if you did that).