Setting up ASE (SQL Server) means creating the system databases, tempdb, "sa" and proble logins and so on. Once you have created the ASE via "sybsetup" or "srvbuild" you will need to create additional databases, extending tempdb (initially at 2MB by default) and creating new logins on the server. You may require to set database options on your sybsystemdb database and dbccdb (if you have installed) it. You may require to set up database options on user databases of course. On memory management you will have to increase the total or max memory of SQL Server (in proportion to max shared memory you have set up on the host) and adjust the percent or actual amount of procedure cache, create 2K, 4K and 16K pool configuration, increase the numbers of locks (for row level locking, and number of open indexes and open objects and so on.
Hope this helps and good luck