Selected Problems from
A First Course in Probability by Sheldon Ross

Problem 27 (Page 253)

f[
          x_] := (1/Beta)[a, 
                b] (x^((a - 
                  1))) (((1 - x))^((b - 1)));

xStar = (((a - 1))/(a + b - 2));

In part a and b, to check if the second derivative is negative we must compute it.  We obtain

D[f[x], {x, 2}] /. {x -> xStar} // Simplify

For part a, if we assume that a>1 and b>1, then a+b>2 and the above is negative, while for part b if we assume that a<1 and b<1, then a+b<2 and the above is positive implying a minimum at the point xStar.