Difference between revisions of "File System and I/O dos and don'ts"
Jump to navigation
Jump to search
(Created page with "====Dos==== * Make a data plan. * Regularly check disk usage with /scinet/gpc/bin/diskUsage. * RAM is always faster than disk: ramdisk. * Write binary files. * Use parallel I/O...") |
(→Dos) |
||
Line 2: | Line 2: | ||
* Make a data plan. | * Make a data plan. | ||
− | * Regularly check disk usage with /scinet/gpc/bin/diskUsage. | + | * Regularly check disk usage with /scinet/gpc/bin/diskUsage, which is updated hourly. |
* RAM is always faster than disk: ramdisk. | * RAM is always faster than disk: ramdisk. | ||
* Write binary files. | * Write binary files. |
Latest revision as of 09:55, 5 October 2010
Dos
- Make a data plan.
- Regularly check disk usage with /scinet/gpc/bin/diskUsage, which is updated hourly.
- RAM is always faster than disk: ramdisk.
- Write binary files.
- Use parallel I/O if writing from dozens of nodes.
- Use gzip and tar.
- Delete unneeded files.
- Maximize size of files.
- Do housekeeping regularly.
- Monitor disk actions with top and strace.
- Attend parallel I/O course coming soon!
- Make an appt to talk with our analysts about your I/O.
Don'ts
- Do not write lots of ASCII files.
- Do not write many hundreds of files in a single directory.
- Do not write many small files.
- Minimize use of file system commands like ls and du.
More information
See the slides of the TechTalk Parallel File System and I/O.