The War Lord (1965)

A knight in the service of a duke goes to a coastal villiage where an earlier attempt to build a defensive castle has failed. He begins to rebuild the duke's authority in the face of the barbarians at the border and is making progress until he falls in love with one of the local women.