First, you'll need to find the marginal distributions of [tex]X,Y[/tex]. By the law of total probability,
[tex]P(X=x)=\displaystyle\sum_yP(X=x)P(Y=y)[/tex]
which translates to
[tex]f_X(x)=\displaystyle\int_x^1f_{X,Y}(x,y)\,\mathrm dy=\begin{cases}2(1-x)&\text{for }0<x<1\\0&\text{otherwise}\end{cases}[/tex]
Similarly,
[tex]f_Y(y)=\displaystyle\int_0^yf_{X,Y}(x,y)\,\mathrm dx=\begin{cases}2y&\text{for }0<y<1\\0&\text{otherwise}\end{cases}[/tex]
Compute the expectations for both random variables:
[tex]E[X]=\displaystyle\int_{-\infty}^\infty x\,f_X(x)\,\mathrm dx=\int_0^12x(1-x)\,\mathrm dx=\frac13[/tex]
[tex]E[Y]=\displaystyle\int_{-\infty}^\infty y\,f_Y(y)\,\mathrm dy=\int_0^12y^2\,\mathrm dy=\frac23[/tex]
Compute the variances and thus standard deviations:
[tex]V[X]=E[(X-E[X])^2]=E[X^2]-E[X]^2[/tex]
where
[tex]E[X^2]=\displaystyle\int_{-\infty}^\infty x^2\,f_X(x)\,\mathrm dx=\int_0^12x^2(1-x)\,\mathrm dx=\frac16[/tex]
[tex]\implies V[X]=\dfrac16-\left(\dfrac13\right)^2=\dfrac1{18}\implies\sqrt{V[X]}=\dfrac1{3\sqrt2}[/tex]
[tex]E[Y^2]=\displaystyle\int_{\infty}^\infty y^2f_Y(y)\,\mathrm dy=\int_0^12y^3\,\mathrm dy=\frac12[/tex]
[tex]\implies V[Y]=\dfrac12-\left(\dfrac23\right)^2=\dfrac1{18}\implies\sqrt{V[Y]}=\dfrac1{3\sqrt2}[/tex]
Compute the covariance:
[tex]\operatorname{Cov}[X,Y]=E[(X-E[X])(Y-E[Y])]=E[XY]-E[X]E[Y][/tex]
We have
[tex]E[XY]=\displaystyle\int_{-\infty}^\infty\int_{-\infty}^\infty xy\,f_{X,Y}(x,y)\,\mathrm dx\,\mathrm dy=\int_0^1\int_0^y2xy\,\mathrm dx\,\mathrm dy=\frac14[/tex]
and so
[tex]\operatorname{Cov}[X,Y]=\dfrac14-\dfrac13\dfrac23=\dfrac1{36}[/tex]
Finally, the correlation:
[tex]\operatorname{Corr}[X,Y]=\dfrac{\operatorname{Cov}[X,Y]}{\sqrt{V[X]}\sqrt{V[Y]}}=\dfrac{\frac1{36}}{\left(\frac1{3\sqrt2}\right)^2}=\dfrac12[/tex]
