The short answer is you try and allocate it. If it succeeds, carry on and do whatever you want to do. If it fails, print an error message and quit.
If your matrix (and your code) can be reasonably made to function on subsets of the matrix (say by splitting into quarters), there is not a lot else you can do.
> Can I make use of something like RLIMIT_DATA, etc?
IIRC, these are theoretical limits based on some ideal. They're not representative of what your machine is capable of at that moment in time.
For example, my "ulimit -d" prints "unlimited", which is of course a nonsense in any real universe.
--