Missing Memory on Solaris ? ZFS is the explanation

By | October 15, 2013

Okay – ZFS ‘May be’ the explanation to your missing memory !!!! But if you have Solaris server running ZFS and you can not explain memory consumption then this most likely applies to you.

Solaris / ZFS Missing Memory

After support team installs or migrates their application to new Solaris servers using ZFS file system, they may notice couple of things

1. Some of the memory is missing on Solaris server

When I say missing, I mean that the total memory consumed by all the applications running and total used memory on the server do not tally. Unix Administrators can generally check memory utilization of server easily using commands/utilities like ‘top'

2. Application start up / restarts are taking less time than earlier

In some cases these may be as less as a tenth of earlier time.

Are these observations linked to each other? – Yes. Administrators can run following command to see what’s going on. It needs root access to run this command

MySevrer $ echo ::memstat | mdb –k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    1233530              9636   10%
ZFS File Data             5697227             44509   46%
Anon                      3528222             27564   29%
Exec and libs               12173                95    0%
Page cache                 247138              1930    2%
Free (cachelist)           125278               978    1%
Free (freelist)           1514212             11829   12%
Total                    12357780             96545
Physical                 12334460             96362

As you can see from the output that ZFS file data is occupying about 46% of total memory which is close to 43 GB. Why does a file system (ZFS) needs such a large memory (RAM) ?
Answer lies in ZFS’ implementation of Disk Cache which uses ARC (Adaptive Replacement Cache). ZFS will try to keep as much disk data in cache (RAM) as possible. Influencing factors like availability of unused RAM and ARC configurations/setting will decide how much of Memory (RAM) should be utilized for Disk Cache.
If Memory is large enough, then Applications can directly work out of disk cache most of the time, without touching the disk itself (or rarely touching the disk, to be more realistic).
Thus applications speed up considerably as reading from and writing to RAM is much faster than reading from and writing to disk.

Will the large memory occupied by ZFS cause problems ?

Obvious question arises that what if some or most of that 43 GB memory occupied by ZFS is required by Application(s) in future ? Well ZFS and ARC manage this memory intelligently. They will use the memory for maintaining disk cache, only if it is lying unused. If the any other application or OS requires memory, ZFS will release the memory used by for disk cache.


How to tune ARC or Memory consumed by ZFS ?

ARC configurations can be tuned to set maximum limit on memory that should be utilized for disk cache. To set this value you can either use /etc/system file OR set zfs:zfs_arc_max command

To check current ARC configuration and usage use following command.

MySevrer $ kstat -p zfs:0:arcstats
zfs:0:arcstats:c        52683216910
zfs:0:arcstats:c_max    100161191936            #