You might receive this error if the LD_LIBRARY_PATH variable isn’t set properly and run MATLAB directly without using the script (dsp_builder.sh).
Since version 11.0 onward, the correct way to start MATLAB with DSP Builder is to use the quartus/dsp_builder/dsp_builder.sh script. This script requires no modifications to MATLAB and it allow user to use more than one version of DSP Builder and more than one version of MATLAB without re-installation.
You can use the following options after the dsp_builder.sh command:
1) -m <path to MATLAB> to specify another MATLAB path.
2) -glnx86 to run 32-bit DSP Builder