P=\frac{3(x^2+y^2)+3(x+y)}{4}+\frac{3-(x+y)}{x+y}-(x^2+y^2)=-\frac{x^2+y^2}{4}+\frac{3(x+y)}{4}+\frac{3}{x+y}-1
=-\frac{(x+y)^2}{4}+(x+y)-1-\frac{x+y}{4}+\frac{3}{x+y}+\frac{xy}{2}
=-(\frac{x+y}{2}-1)^2-\frac{x+y}{4}+\frac{3}{x+y}+\frac{3-(x+y)}{2}
=-(\frac{x+y}{2}-1)^2-3(\frac{x+y}{4}-\frac{1}{x+y})+\frac{3}{2}
=-(\frac{x+y}{2}-1)^2-3.\frac{(x+y)^2-4}{4(x+y)}+\frac{3}{2} (1)
Ta có: 3=xy+x+y \le \frac{(x+y)^2}{4}+x+y=(\frac{x+y}{2}+1)^2-1
=> (\frac{x+y}{2}+1)^2 \ge 4
=> x+y \ge 2
=> (x+y)^2 \ge 4 (do x+y>0) (2)
từ (1) và (2) => P \le \frac{3}{2}
Dấu = xảy ra <=> x=y=1
Vậy P_{max}=\frac{3}{2} tại x=y=1