Nice problem!
Area of a rectangle = length * width, or length = Area / width
Let the Width be W
then the length is 832/W
Assume the side W is a fence, and the rest brick.
Total cost, C(W) = $5*W + $8W + 2*$8*(832/W)
Simplifying, C(W)=13W+13312/W
To find the minimum cost, we differentiate the cost with respect to W, and equate the derivative C'(W) to zero.
then C'(W)=13-13312/W^2=0
Rewrite C'(W) as (13W^2-13312)/(W^2)=0, we solve for W and get
W=sqrt(13312/13)=32
Therefore length, L=832/32=26
Check L*W=26*32=832 ok [ note L>W, because this costs less $]
Check L=26 and W=32 is the minimum (as opposed to maximum),
we calculate C"(W)=26624/W^3 > 0 which means that W=26 is a minimum for C(W).
So the dimensions of the garden are 26' x 32', with fence on the W=32' dimension.
Just by curiosity, total cost = C(32)=$832, and average cost = $7.17/'
all sound reasonable.