Difference between revisions of "BGQ"
| Line 25: | Line 25: | ||
BGQ job size is typically determined by midplanes (512 nodes or 8192 cores), however sub-blocks can be used to further subdivide midplanes with a minimum of | BGQ job size is typically determined by midplanes (512 nodes or 8192 cores), however sub-blocks can be used to further subdivide midplanes with a minimum of | ||
| − | one IO node per block. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest job size. | + | one IO node per block. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest job size. A block is setup to match a jobs configuration (number of nodes and and network configuration) and partitioned and booted specifically for that job. If a job with the same requirements matches an existing running job then the nodes are not required to be rebooted/repartitioned, however if the any of the parameters are different the block needs to be recreated and rebooted to match the new jobs specifications. When running with loadleveler this happens automatically, and due to the lightweight OS used on the compute nodes, happens very quickly. |
Revision as of 15:49, 21 August 2012
| Blue Gene/Q (BGQ) | |
|---|---|
| Installed | August 2012 |
| Operating System | RH6.3, CNK (Linux) |
| Number of Nodes | 2048(32,768 cores), 512 (8,192 cores) |
| Interconnect | 5D Torus (jobs), QDR Infiniband (I/O) |
| Ram/Node | 16 Gb |
| Cores/Node | 16 (64 threads) |
| Login/Devel Node | bgq01,bgq02 |
| Vendor Compilers | bgxlc, bgxlf |
| Queue Submission | Loadleveler |
Specifications
BGQ is an extremely powerful and energy efficient 3rd generation IBM Supercomputer built around a system on a chip compute node that has a 16core 1.6GHz Power based CPU and 16Gb of Ram and runs a very lightweight Linux OS called CNK. The nodes are bundled in groups of 32 and then 16 of these groups make up a midplane with 2 midplanes per rack. The compute nodes are all connected togther using a custom 5D interconnect. Each midplane has 8 Power7 I/O nodes that run a full Redhat Linux OS that manages the compute nodes and mounts the GPFS filesystem.
Jobs
BGQ job size is typically determined by midplanes (512 nodes or 8192 cores), however sub-blocks can be used to further subdivide midplanes with a minimum of one IO node per block. In SciNet's configuration (with 8 I/O nodes per midplane) this allows 64 nodes (1024 cores) to be the smallest job size. A block is setup to match a jobs configuration (number of nodes and and network configuration) and partitioned and booted specifically for that job. If a job with the same requirements matches an existing running job then the nodes are not required to be rebooted/repartitioned, however if the any of the parameters are different the block needs to be recreated and rebooted to match the new jobs specifications. When running with loadleveler this happens automatically, and due to the lightweight OS used on the compute nodes, happens very quickly.
*** WAT2Q SPECIFIC ****
Compile
/bgsys/drivers/V1R1M1/ppc64/comm/xl/bin/mpich2version /bgsys/drivers/V1R1M1/ppc64/comm/xl/bin/mpixlc /bgsys/drivers/V1R1M1/ppc64/comm/xl/bin/mpixf90
Run a Job
When not using loadleveler there is a direct launch program called runjob on BGQ that acts a lot like mpirun/mpiexec. The "block" argument is the predifined group of nodes that are already booted. See the next section on how to create these blocks manually. Note that a block does not need to be rebooted between jobs, only if the number of nodes or network parameters are need to be changed.
runjob --block R00-M0-N03-32 --ranks-per-node=16 --np 512 --cwd=/gpfs/DDNgpfs3/xsnorthrup/osu_bgq --exe=/gpfs/DDNgpfs3/xsnorthrup/osu_bgq/osu_mbw_mr
To see running jobs and the status of available blocks use:
list_jobs list_blocks
Setup blocks
To reconfigure the BGQ nodes use the bg_console
bg_console
There are various options to create block types (section 3.2 in the BGQ admin manual), but the smallest is created using the following command:
gen_small_block <blockid> <midplane> <cnodes> <nodeboard> gen_small_block R00-M0-N03-32 R00-M0 32 N03
The block then needs to be booted using:
allocate R00-M0-N03-32
If those resources are already booted into another block, that block must be freed before the new block can be allocated.
select_block R00-M0-N03 free_block
There are many other functions in bg_console:
help all
I/O
GPFS