Stale Native Code Files Are Being Cached with File Names Such as: JOXSHM_EXT*, PESHM_EXT*, PESLD* or SHMDJOXSHM_EXT*

JOXSHM_EXT and *EXT* files created under /dev/shm :

Files with various name formats are being created but not removed which may lead to file storage issues or errors. File name formats are similar to the following:

JOXSHM_EXT_*_sid_name_*
and / or
PESHM_EXT_*_sid_name_*
and / or
PESLD_sid_name_*
and / or
.SHMDJOXSHM_EXT_*_sid_name_*
and / or
.SHMDPESHM_EXT_*_sid_name_*
and / or
.SHMDPESLD_sid_name_*

The location of these files depends on the operating system and how it implements shm_open() there. On platforms such as Linux and Solaris object based shared memory segments are used and the files are found in /dev/shm and /tmp respectively. On AIX, no files are seen at the file system level, however the shared memory segments can be seen using the 'ipcs -ar' command. On other platforms such as HP Itanium, file based shared memory segments are used, with the default location for HP being $ORACLE_HOME/dbs. 

When stale copies of these files are not correctly cleaned up this can impact the server's shared memory and paging space, requiring a server reboot to clear the memory.

This issue is seen with 11.1 or higher.

The files are created when PL/SQL and Java code is natively compiled in the database. Following a normal database shutdown one would expect Oracle to clean up these shared memory files, but if an instance fails for any reason and/or is not closed cleanly then the files are left behind. For example the SHUTDOWN ABORT command does not remove these cached native code files.

This is documented in the 11.2 Readme.

These shared memory segments may also remain following a normal shutdown of a RAC database prior to 12.1.0.2.

NOTE: DO NOT delete the files while the database is up and running as it will cause ORA-7445 / ORA-600 errors, other unexpected errors and adverse affects on the database.

SOLUTION :

Copies of these files for a given database instance should be automatically removed when the database is shut down normally. Stale files relating to code that has aged out of the shared pool are also removed when new or existing code is (re)compiled or (re)loaded into the shared pool.

Pre-12.1.0.2 RAC database is shut down normally but native cache files are not being removed

Apply a patch for one or both of the following bugs as appropriate:

a) For JavaVM related files: Bug 18128196 NATIVE COMPILATION FILES ARE NOT GETTING CLEARED IN RAC AFTER DATABASE SHUTDOWN

b) For PLSQL related files: Unpublished bug 18800913 COMPILE ISSUE IN KKXWTP ON NON-BASE PLATFORMS (this supersedes the original fix done in bug 18327244.)

Database has not been shut down normally so native cache files remain

Manually deleting these types of files should only ever be done when the database is down. Once located the files can be removed via the appropriate OS command. On AIX these files are removed using the 'ipcrm' command.

To make it easier to find any files to be deleted, there is a parameter which controls their location, _ncomp_shared_objects_dir. This was introduced in 11.1.0.7.8 and 11.2.0.2 for use with PL/SQL native files on all platforms and Java native files on platforms that use file based shared memory segments. For platforms using object based shared memory segments, such as Linux, Solaris and AIX, the parameter does not have any effect for Java cached native code files until 11.2.0.4 and 12c where the fix for bug 13574534 is present. 

To use this method:

1. Set _ncomp_shared_objects_dir, either in your init.ora file, or via alter system with scope=spfile before you shut down the database. Use a unique directory name for each instance. Do not use that directory for anything else so you can remove it any time the instance is down.

NOTE: The underscore parameter does not prevent the creation of the files, but does allow you to write them to a known location so that they can be manually removed as the database is restarted.

2. Each time the database is shut down, rename the directory and create a new one with the old name then remove the renamed directory.

If multiple databases exist then each should have a different location specified. This makes it easier to clean up left over files when the database is down with no possibility of removing files being used by a running instance.