Ta có:
$\sin^2A+\sin^2B+\sin^2C$
$=2-\dfrac{1}{2}(\cos2A+\cos2B)-\cos^2C$
$=2-\cos(A+B)\cos(A-B)-\cos^2C$
$=2+\cos C[\cos(A-B)-\cos C]$
$\le2+\cos C(1-\cos C)$
$\le2+\left[\dfrac{\cos C+(1-\cos C)}{2}\right]^2=\dfrac{9}{4}$
Từ đó suy ra: $\cos^2A+\cos^2B+\cos^2C\ge\dfrac{3}{4}$
Áp dụng BĐT Cauchy ta có:
$\cos^3A+\cos^3A+\dfrac{1}{8}\ge\dfrac{3}{2}\cos^2A$
$\cos^3B+\cos^3B+\dfrac{1}{8}\ge\dfrac{3}{2}\cos^2B$
$\cos^3B+\cos^3B+\dfrac{1}{8}\ge\dfrac{3}{2}\cos^2C$
Suy ra: $\cos^3A+\cos^3B+\cos^3C\ge\dfrac{3}{8}$
$\Rightarrow A\le\dfrac{9\alpha}{4}-\dfrac{3\beta}{8}$
Vậy: $\max A=\dfrac{9\alpha}{4}-\dfrac{3\beta}{8} \Leftrightarrow A=B=C=\dfrac{\pi}{3}$