It looks like this is a system of linear ODEs given in matrix form,
[tex]x' = \begin{bmatrix}10&-1\\5&8\end{bmatrix} x[/tex]
with initial condition x(0) = (-6, 8)ᵀ.
Compute the eigenvalues and -vectors of the coefficient matrix:
[tex]\det\begin{bmatrix}10-\lambda&-1\\5&8-\lambda\end{bmatrix} = (10-\lambda)(8-\lambda) + 5 = 0 \implies \lambda^2-18\lambda+85=0 \implies \lambda = 9\pm2i[/tex]
Let v be the eigenvector corresponding to λ = 9 + 2i. Then
[tex]\begin{bmatrix}10-\lambda&-1\\5&8-\lambda\end{bmatrix}v = 0 \implies \begin{bmatrix}1-2i&-1\\5&-1-2i\end{bmatrix}\begin{bmatrix}v_1\\v_2\end{bmatrix}=\begin{bmatrix}0\\0\end{bmatrix}[/tex]
or equivalently,
[tex]\begin{cases}(1-2i)v_1-v_2=0 \\ 5v_1-(1+2i)v_2=0\end{cases} \implies 5v_1 - (1+2i)v_2 = 0[/tex]
Let [tex]v_2=1-2i[/tex]; then [tex]5v_1-5=0\implies v_1=1[/tex], so that
[tex]\begin{bmatrix}10&-1\\5&8\end{bmatrix}\begin{bmatrix}1\\1-2i\end{bmatrix} = (9+2i)\begin{bmatrix}1\\1-2i\end{bmatrix}[/tex]
and we get the other eigenvalue/-vector pair by taking the complex conjugate,
[tex]\begin{bmatrix}10&-1\\5&8\end{bmatrix}\begin{bmatrix}1\\1+2i\end{bmatrix} = (9-2i)\begin{bmatrix}1\\1+2i\end{bmatrix}[/tex]
Then the characteristic solution to the system is
[tex]x = C_1 e^{(9+2i)t} \begin{bmatrix}1\\1-2i\end{bmatrix} + C_2 e^{(9-2i)t} \begin{bmatrix}1\\1+2i\end{bmatrix}[/tex]
From the given condition, we have
[tex]\displaystyle \begin{bmatrix}-6\\8\end{bmatrix} = C_1 \begin{bmatrix}1\\1-2i\end{bmatrix} + C_2 \begin{bmatrix}1\\1+2i\end{bmatrix} \implies C_1 = -3-\frac i2, C_2=-3+\frac i2[/tex]
and so the particular solution to the IVP is
[tex]\displaystyle \boxed{x = -\left(3+\frac i2\right) e^{(9+2i)t} \begin{bmatrix}1\\1-2i\end{bmatrix} - \left(3-\frac i2\right) e^{(9-2i)t} \begin{bmatrix}1\\1+2i\end{bmatrix}}[/tex]
which you could go on to rewrite using Euler's formula,
[tex]e^{(a+bi)t} = e^{at} (\cos(bt) + i \sin(bt))[/tex]
