How often should I reboot my Linux server?

We recommend that you reboot your Linux server every month to install kernel updates from Red Hat, firmware upgrades from the server’s hardware vendor, and perform low-level system integrity checks. You may certainly reboot as often as your maintenance schedule allows, but it is not required for the stable operation of Eclipse. For more information, see: How often should I update my Red Hat Enterprise Linux server?

Expand Linux Filesystem

If you need to expand or add space to a filesystem under Linux, please follow the procedure outlined below, replacing the directory and filesystem names with your own:

Verify the current filesystem usage and device path:

df -h /u2/eclipse

Verify that the volume group for sufficient space:

vgs datavg

Set the new logical volume size:

lvextend -L 80G /dev/datavg/eclipse

Resize the filesystem:

resize2fs /dev/datavg/eclipse

Verify the updated filesystem utilization:

df -h /u2/eclipse

Here’s an example:

[root@eclipse ~]# df -h /u2/eclipse
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/datavg-eclipse
                       20G   18G  835M  96% /u2/eclipse
[root@eclipse ~]# vgs datavg
  VG     #PV #LV #SN Attr   VSize   VFree
  datavg   1   5   0 wz--n- 474.22G 406.22G
[root@eclipse ~]# lvextend -L 40G /dev/datavg/eclipse
  Extending logical volume eclipse to 40.00 GB
  Logical volume eclipse successfully resized
[root@eclipse ~]# resize2fs /dev/datavg/eclipse
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/datavg/eclipse is mounted on /u2/eclipse; on-line resizing required
Performing an on-line resize of /dev/datavg/eclipse to 10485760 (4k) blocks.
The filesystem on /dev/datavg/eclipse is now 10485760 blocks long.

[root@eclipse ~]# df -h /u2/eclipse
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/datavg-eclipse
                       39G   18G   20G  48% /u2/eclipse

If the LVM snapshot filesystems are mounted, please follow the procedure outlined below.

First, unmount the snap filesystems (and any “child” filesystems, such as /u2/eclipse/ereports under /u2/eclipse):

umount /snap/20110909.1043/u2/eclipse/

Remove the snap logical volumes (which you can identify with the “lvs” command):

lvremove /dev/datavg/lvol1

Set the new logical volume size:

lvextend -L 250G /dev/datavg/eclipse

Resize the filesystem:

resize2fs /dev/datavg/eclipse

If you are unable to unmount or lvremove a snapshot, verify that there are no processes holding the volume open.

How do filesystem snapshots work on Linux?

To perform valid backups of your database, it is important to suspend the database. This prevents modifications of files during the backup process. By taking a point-in-time snapshot of your database files, your backup program will be capturing a “frozen” database instead of an “in motion” database.

Our standard backup script uses database suspension with snapshots to create point-in-time images of your database files. The snapshot script itself is located at /u2/UTILS/bin/snapsave_linux.sh (with a symbolic link at /bin/save for backwards compatibility).

The snapshot script is typically scheduled to run at regular intervals via crontab to create new filesystem snapshots.  Here’s an example of a snapshot backup script that is scheduled via crontab to run every night at 12:59 AM:

[root@eclipse ~]# crontab -l
59 0 * * * /u2/UTILS/bin/snapsave_linux.sh

After running the script, the snapshot filesystems are mounted under /snap, allowing read-only access by backup software. For example, the snapshot of the /u2/eclipse/LEDGER file would be located at /snap/u2/eclipse/LEDGER. When configuring backup software, it is recommended to backup every file under /snap/u2.

Since every change (delta) between the snapshot and the “live” filesystem must be recorded, the snapshots have a finite lifespan. By default, the snapshot script is configured to hold 1GB of changes before requiring a refresh. On busier systems, or on systems where the snapshots must be retained for a longer period of time to accommodate a slow backup process, the snapshot volume size may be increased by editing the snapshot backup script. You may check the status of the snapshots using the “lvs” command, which shows a usage percentage for each snapshot volume.

[root@eclipse ~]# lvs
  LV       VG     Attr   LSize   Origin   Snap%  Move Log Copy%  Convert
  eclipse  datavg owi-ao  26.00G
  ereports datavg owi-ao   1.00G
  lvol0    datavg swi-ao   1.00G u2        45.85
  lvol4    datavg swi-ao   1.00G uvtmp      0.00
  lvol5    datavg swi-a-   1.00G ereports   0.00
  lvol6    datavg swi-ao   1.00G eclipse    0.85
  u2       datavg owi-ao   4.00G
  uvtmp    datavg owi-ao   4.00G
  esupport rootvg -wi-ao   6.00G
  root     rootvg -wi-ao  20.00G
  swap     rootvg -wi-ao   4.00G

When the Snap% value reaches 100%, the snapshot volume has reached its maximum capacity for tracking changes and must be recreated by running the snapshot script again.

For troubleshooting purposes, a log of the snapshot backup script is kept at /tmp/snapsave.log. Information regarding the creation, removal and expiration of snapshot LVs is also recorded in the system log (/var/log/messages).

How do I check the status of my EVault backup?

To manually check the status of your EVault backup:

  • If it’s not already installed, download and install the EVault CentralControl client from EVault’s support website or using this direct link.
  • If you’re using CentralControl for the first time, you’ll need to add connections to each server being backed up by EVault:
    • File -> New Agent
    • Fill in the Description, Network address, and root/Administrator user name and password fields
    • Press OK
  • Click the “+” sign next to the server you wish to check to expand and display the backup job(s)
  • Click the “+” sign next to the backup job you wish to check to expand and display the Logs folder
  • Select the Logs folder and double-click the most recent log to display the log. A successful backup will have no errors with output similar to the following:
06-Sep 21:03 SSET-I-04131 disconnect from the Vault at 06-Sep-2010 21:03:24 -0700
06-Sep 21:03 BKUP-I-00001 errors encountered:                        0
06-Sep 21:03 BKUP-I-00002 warnings encountered:                      0
06-Sep 21:03 BKUP-I-00003 files/directories examined:                280,340
06-Sep 21:03 BKUP-I-00004 files/directories filtered:                0
06-Sep 21:03 BKUP-I-00006 files/directories deferred:                0
06-Sep 21:03 BKUP-I-00007 files/directories backed-up:               280,340
06-Sep 21:03 BKUP-I-00008 files backed-up:                           267,837
06-Sep 21:03 BKUP-I-00009 directories backed-up:                     12,503
06-Sep 21:03 BKUP-I-00010 data stream bytes processed:               14,782,268,676 (13.76 GB)
06-Sep 21:03 BKUP-I-00011 all stream bytes processed:                14,782,268,676 (13.76 GB)
06-Sep 21:03 BKUP-I-00012 pre-delta bytes processed:                 651,912,316 (621.71 MB)
06-Sep 21:03 BKUP-I-00013 deltized bytes processed:                  118,090,710 (112.62 MB)
06-Sep 21:03 BKUP-I-00014 compressed bytes processed:                34,164,618 (32.58 MB)
06-Sep 21:03 BKUP-I-00015 approximate bytes deferred:                0 (0 bytes)
06-Sep 21:03 BKUP-I-00016 reconnections on recv fail:                0
06-Sep 21:03 BKUP-I-00017 reconnections on send fail:                0
06-Sep 21:03 SYST-I-07035 send e-mail: OK
06-Sep 21:03 BKUP-I-04128 job completed at 06-Sep-2010 21:03:26 -0700
06-Sep 21:03 BKUP-I-04129 elapsed time 00:03:11

Here’s a screencast showing an overview of the process:

How do I check disk usage on my Linux server?

View a step-by-step screencast of this process:

Log into the RHEL server as root via PuTTY, or shell out from Eterm.

The “df” command is used to determine overall filesystem usage. The “vgs” command will show you any space available in the volume group that has not yet been allocated to a specific filesystem.

To see the total size of each filesystem:

df -h

The standard Eclipse filesystems are located under /u2. Add the “Used” column for each of these filesystems, plus any non-standard Eclipse filesystems that you may have (/train, /backup, etc.).

[root@firestorm ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-root
                       20G   18G  1.1G  95% /
/dev/mapper/datavg-u2
                      3.9G  2.1G  1.6G  57% /u2
/dev/mapper/datavg-uvtmp
                      3.9G  137M  3.6G   4% /u2/uvtmp
/dev/mapper/rootvg-esupport
                      5.9G  3.4G  2.3G  60% /esupport
/dev/sda1              99M   26M   69M  28% /boot
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/mapper/datavg-eclipse
                       26G   17G  7.6G  69% /u2/eclipse
/dev/mapper/datavg-ereports
                      992M   34M  908M   4% /u2/eclipse/ereports
/dev/mapper/datavg-lvol0
                      3.9G  2.1G  1.6G  57% /snap/u2
/dev/mapper/datavg-lvol4
                      3.9G  137M  3.6G   4% /snap/u2/uvtmp
/dev/mapper/datavg-lvol6
                       26G   17G  7.6G  69% /snap/u2/eclipse

If you would like to identify which files or subdirectories are using the most space in a particular filesystem or directory, you may use the “du” command. For example, to list the disk usage of top files and directories under /u2/eclipse in megabytes:

[root@firestorm ~]# cd /u2/eclipse
[root@firestorm eclipse]# du -sm * | sort -rn | head
3780    modules-linux
2856    modules
1568    ZIP4
859     jsdata
740     PHANTOM.LOG
740     modules-test
450     8.6.9.00.16-aix.zip
447     8.6.9.00.99-aix.zip
446     8.6.9.00.11-aix.zip
428     8.6.9.00.13-aix.zip

If you would like to see how much space is available for allocation within the volume group:

vgs

For more detail on this process, see: How do I expand a Linux file system?

Alternatively, if you prefer a graphical tool, you may log into the server’s GUI, and use the Disk Usage Analyzer tool:

  • Log into the server’s GUI (see also How do I access the GUI of my Linux server?)
  • Applications -> System Tools -> Disk Usage Analyzer
  • Click the Filesystem button to begin storage analysis of your entire system
  • Use the controls to drill down into each subdirectory for a graphical picture of storage utilization