On Solaris 10 there are a number of resource limits that can be imposed on a Solaris 10 project.
The “prjstat -J” command gives you a nice basic summary of overall project resources being used. ”
PROJID NPROC SWAP RSS MEMORY TIME CPU PROJECT
3 350 6106M 2677M 16% 22:41:16 2.2% default
1 4 2704K 4096K 0.0% 0:08:10 0.8% user.root
0 45 198M 225M 1.4% 0:17:25 0.1% system
100 13 121M 67M 0.4% 0:24:27 0.0% group.mqmSo from the output above you can see that I have 4 projects defined. On my system, of particular note is the fact that other than for members of group mqm everyone uses the “default” project to define limits system wide. So how do we find out what limits are currently defined?
The “prctl” command will give you the limits currently imposed for a particular process. Once again because everyone uses the “default” project simply sign-in as a normal user and execute the “prctl” command with “$$” as arguments to use the current shell’s process to determine resource limits.
dev:/> prctl $$ process: 20496: -ksh NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT process.max-port-events privileged 65.5K - deny - system 2.15G max deny - process.max-msg-messages privileged 8.19K - deny - system 4.29G max deny - process.max-msg-qbytes privileged 64.0KB - deny - system 16.0EB max deny - process.max-sem-ops privileged 512 - deny - system 2.15G max deny - process.max-sem-nsems privileged 512 - deny - system 32.8K max deny - process.max-address-space privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-descriptor privileged 8.19K - deny - system 2.15G max deny - process.max-core-size privileged 8.00EB max deny - system 8.00EB max deny - process.max-stack-size basic 8.00MB - deny 20496 privileged 8.00EB - deny - system 8.00EB max deny - process.max-data-size privileged 16.0EB max deny - system 16.0EB max deny - process.max-file-size privileged 8.00EB max deny,signal=XFSZ - system 8.00EB max deny - process.max-cpu-time privileged 18.4Es inf signal=XCPU - system 18.4Es inf none - task.max-cpu-time system 18.4Es inf none - task.max-lwps system 2.15G max deny - project.max-contracts privileged 10.0K - deny - system 2.15G max deny - project.max-device-locked-memory privileged 996MB - deny - system 16.0EB max deny - project.max-locked-memory system 16.0EB max deny - project.max-port-ids privileged 8.19K - deny - system 65.5K max deny - project.max-shm-memory privileged 3.89GB - deny - system 16.0EB max deny - project.max-shm-ids privileged 128 - deny - system 16.8M max deny - project.max-msg-ids privileged 128 - deny - system 16.8M max deny - project.max-sem-ids privileged 128 - deny - system 16.8M max deny - project.max-crypto-memory privileged 3.89GB - deny - system 16.0EB max deny - project.max-tasks system 2.15G max deny - project.max-lwps system 2.15G max deny - project.cpu-cap system 4.29G inf deny - project.cpu-shares privileged 1 - none - system 65.5K max none - zone.max-swap system 16.0EB max deny - zone.max-locked-memory system 16.0EB max deny - zone.max-shm-memory system 16.0EB max deny - zone.max-shm-ids system 16.8M max deny - zone.max-sem-ids system 16.8M max deny - zone.max-msg-ids system 16.8M max deny - zone.max-lwps system 2.15G max deny - zone.cpu-cap system 4.29G inf deny - zone.cpu-shares privileged 1 - none - system 65.5K max none -
A threshold value on a resource control constitutes a point at which local actions can be triggered or global actions, such as logging, can occur.
Each threshold value on a resource control must be associated with a privilege level. The privilege level must be one of the following three types:
- basic
- Can be modified by the owner of the calling process.
- privileged
- Can be modified by the current process (requiring sys_resource privilege) or by prctl(1) (requiring proc_owner privilege).
- system
- Fixed for the duration of the operating system instance.
A resource control is guaranteed to have one system value, which is defined by the system, or resource provider. The system value represents how much of the resource the current implementation of the operating system is capable of providing.
Any number of privileged values can be defined, and only one basic value is allowed. Operations that are performed without specifying a privilege value are assigned a basic privilege by default.
So how do we know if a particular limit has been hit? Usually it is obvious because there is a long line of users lined up outside your office because they are receiving nasty system messages when trying to do something. In my case, a whole bunch of developers were complaining that the Micro Focus COBOL compiler was dumping out the error:
DIRECT: Memory allocation failed
This kind of obscure message is usually your first clue that a resource limit of some kind has been hit. Many times, the other parts of the system continue to run fine.
Now you would expect that some kind of message would be dumped to syslog or something when a threshold is reached, but on Solaris 10 the default is not to log resource limit threshold notifications. You can see whether they are enabled or not by using the “rctladm” command:
dev/# rctladm process.max-port-events syslog=off [ deny count ] process.max-msg-messages syslog=notice [ deny count ] process.max-msg-qbytes syslog=off [ deny bytes ] process.max-sem-ops syslog=off [ deny count ] process.max-sem-nsems syslog=off [ deny count ] process.max-address-space syslog=off [ lowerable deny no-signal bytes ] process.max-file-descriptor syslog=off [ lowerable deny count ] process.max-core-size syslog=off [ lowerable deny no-signal bytes ] process.max-stack-size syslog=off [ lowerable deny no-signal bytes ] process.max-data-size syslog=off [ lowerable deny no-signal bytes ] process.max-file-size syslog=off [ lowerable deny file-size bytes ] process.max-cpu-time syslog=off [ lowerable no-deny cpu-time inf seconds ] task.max-cpu-time syslog=off [ no-deny cpu-time no-obs inf seconds ] task.max-lwps syslog=off [ count ] project.max-contracts syslog=off [ no-basic deny count ] project.max-device-locked-memory syslog=off [ no-basic deny bytes ] project.max-locked-memory syslog=off [ no-basic deny bytes ] project.max-port-ids syslog=off [ no-basic deny count ] project.max-shm-memory syslog=notice [ no-basic deny bytes ] project.max-shm-ids syslog=notice [ no-basic deny count ] project.max-msg-ids syslog=off [ no-basic deny count ] project.max-sem-ids syslog=off [ no-basic deny count ] project.max-crypto-memory syslog=off [ no-basic deny bytes ] project.max-tasks syslog=off [ no-basic count ] project.max-lwps syslog=off [ no-basic count ] project.cpu-cap syslog=off [ no-basic deny no-signal inf count ] project.cpu-shares syslog=n/a [ no-basic no-deny no-signal no-syslog count ] zone.max-swap syslog=off [ no-basic deny bytes ] zone.max-locked-memory syslog=off [ no-basic deny bytes ] zone.max-shm-memory syslog=off [ no-basic deny bytes ] zone.max-shm-ids syslog=off [ no-basic deny count ] zone.max-sem-ids syslog=off [ no-basic deny count ] zone.max-msg-ids syslog=off [ no-basic deny count ] zone.max-lwps syslog=off [ no-basic count ] zone.cpu-cap syslog=off [ no-basic deny no-signal inf count ] zone.cpu-shares syslog=n/a [ no-basic no-deny no-signal no-syslog count ]
To turn on syslog for resource “project.max-shm-memory” you would enter:
dev/# rctladm -e syslog project.max-shm-memory
You need be root on the global zone (if you use them) to run the command. It modifies the /etc/rctladm.conf file so that changes are preserved across reboots. Now when the limit is hit you’ll at least get something syslog like this:
Jan 8 10:15:15 testmachine unix: [ID 859581 kern.notice] NOTICE: privileged rctl project.max-shm-memory exceeded by process 859581
For me, the most likely culprits for a resource limit to be hit was with IPC message queues or shared memory. I have an application that eats these things up like they are going out of style. So the follow resources are what I need to monitor:
project.max-shm-memory
privileged 3.89GB - deny -
system 16.0EB max deny -
project.max-shm-ids
privileged 128 - deny -
system 16.8M max deny -
project.max-msg-ids
privileged 128 - deny -
system 16.8M max deny -I set them up for logging to syslog using the previous technique but wanted to get a feel for what my current usage was…..
From the output above, we can see that “project.max-shm-memory” has a limit of 3.89GB (the default is 1/4 of the total) and project.max-shm-ids has a limit of 128.
From our prjstat command earlier, we quickly see that the default project is only currently using 16% of my total 16GB of memory, so it doesn’t look like I’ve hit that limit.
PROJID NPROC SWAP RSS MEMORY TIME CPU PROJECT
3 350 6106M 2677M 16% 22:41:16 2.2% default
1 4 2704K 4096K 0.0% 0:08:10 0.8% user.root
0 45 198M 225M 1.4% 0:17:25 0.1% system
100 13 121M 67M 0.4% 0:24:27 0.0% group.mqmTo determine the number of “project.max-shm-ids” we are currently using I can use a series of “ipcs” commands, pipe the output to grep to remove group “mqm” processes (because they are in a project of their own), and then count the lines:
dev:/# ipcs -q|grep -v mqm| wc -l 124 dev:/# ipcs -m|grep -v mqm| wc -l 114
Ouch, both of these look like they are close to the 128 default project limit.
Had a couple of choices here, I could divide the system workload by creating other projects or I could just bump up the values of the resources for the default project. Decided to just double the value for the default project. Even though we hadn’t hit the limit yet I decided to also double the “project.max-shm-memory” value because if I allow more shared memory segments to be created we can expect the 3.98 GB limit to quickly be hit. If I increase one I should increase the other!
This can be done in real-time using “prctl” and then followed with “projmod” to allow the changes to survive scross reboots:
dev:/# prctl -n project.max-msg-ids -v 256 -r -i project default dev:/# prctl -n project.max-shm-ids -v 256 -r -i project default dev:/# prctl -n project.max-shm-memory-ids -v 8GB -r -i project default dev:/# projmod -K 'project.max-msg-ids=(privileged,256,deny);project.max-shm-ids=(privileged,256,deny);project.max-shm-memory=(privileged,8GB,deny)' default
The “projmod” command modifies the actual /etc/project file:
dev:/# cat /etc/project system:0:::: user.root:1:::: noproject:2:::: default:3::::project.max-msg-ids=(privileged,256,deny);project.max-shm-ids=(privileged,256,deny);project.max_shm-memory=(privileged,8GB,deny) group.staff:10:::: group.mqm:100:WebSphere MQ default settings:::process.max-file-descriptor=(basic,10000,deny);project.max-sem-ids=(priv,1024,deny);project.max-shm-ids=(priv,1024,deny);project.max-shm-memory=(priv,4294967296,deny)
For a complete listing of the available resource controls do a man on resource_controls