Most likely only 3 pins are used. Rd, Td and Ground.
On the DB9 those are Rd on pin 2, Td on pin 3 and ground on pin 5.
On the RJ11 those pins are most likely Rd on 2, Td on 3 and ground on pin 1.
If you have a DC voltmeter connect the black lead to a RJ11 on pin 1 and check for voltage on pin 2 or 3. Whichever pin has voltage is the Td pin which should go to pin 2 (Rd) on the DB9. The other pin (2 or 3 on the RJ11) should go to pin 3 (Td) on the DB9.
On the RJ11 plug pin 1 is on the right if you hold the cable in your hand with the pins up and latch away from you.