oldwiki.scinet.utoronto.ca - User contributions [en-gb]

Oldwiki.scinet.utoronto.ca:System Alerts

2018-05-06T21:23:06Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:down.png|up|link=HPSS]][https://docs.scinet.utoronto.ca/index.php/HPSS HPSS]
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]External Network
|
|}

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 30 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

SciNet User Support Library

2018-04-26T16:54:44Z

Northrup: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==
* April 23, 2018: GPC & Sandy decommissioned.
* April 10, 2018: Niagara commissioned.
* March 20th, 2018: Gravity decommissioned.
* Dec 4, 2017 : scratchtcs decommissioned.
* Nov 28, 2017: The GPC has been reduced from 30,912 to 16,800 cores to make room for the installation of Niagara.
* Sept 29, 2017: The TCS has been decommissioned.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [https://courses.scinet.utoronto.ca SciNet education]
* [https://www.youtube.com/channel/UC42CaO-AAQhwqa8RGzE3daQ SciNet YouTube Channel]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
* [[2016_Ontario_Summer_School_for_High_Performance_Computing_Central]]
* [[2015_Ontario_Summer_School_for_High_Performance_Computing_Central]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-26T16:44:57Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]External Network
|
|}

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-26T16:44:39Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|-
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]External Network
|
|}

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-26T16:43:40Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:down.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-24T14:42:30Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:down.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:down.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-23T16:00:20Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:down.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up50.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:down.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 23 Apr 2018 BGQ front-end node is down, hopefully fixed later today.

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-23T16:00:03Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:down.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up50.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:down.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 23 Apr 2018 BGQ front-end node is down, hopefully fixed late today.

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-23T15:19:05Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:down.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:down.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

SciNet User Support Library

2018-04-23T15:18:51Z

Northrup: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==
* April 23, 2018: GPC decommissioned.
* April 10, 2018: Niagara commissioned.
* Dec 4, 2017 : scratchtcs decommissioned.
* Nov 28, 2017: The GPC has been reduced from 30,912 to 16,800 cores to make room for the installation of Niagara.
* Sept 29, 2017: The TCS has been decommissioned.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [https://courses.scinet.utoronto.ca SciNet education]
* [https://www.youtube.com/channel/UC42CaO-AAQhwqa8RGzE3daQ SciNet YouTube Channel]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
* [[2016_Ontario_Summer_School_for_High_Performance_Computing_Central]]
* [[2015_Ontario_Summer_School_for_High_Performance_Computing_Central]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-23T15:18:16Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:down.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-23T15:17:23Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:down.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

GPC-compute will be decommissioned on Sat 21 Apr 2018, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-19T01:11:16Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

GPC-compute will be decommissioned on Sat 21 Apr 2018, GPC-storage available until 9 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

SciNet User Support Library

2018-04-13T01:22:16Z

Northrup: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==
* April 10, 2018: Niagara commissioned.
* Dec 4, 2017 : scratchtcs decommissioned.
* Nov 28, 2017: The GPC has been reduced from 30,912 to 16,800 cores to make room for the installation of Niagara.
* Sept 29, 2017: The TCS has been decommissioned.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [https://courses.scinet.utoronto.ca SciNet education]
* [https://www.youtube.com/channel/UC42CaO-AAQhwqa8RGzE3daQ SciNet YouTube Channel]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
* [[2016_Ontario_Summer_School_for_High_Performance_Computing_Central]]
* [[2015_Ontario_Summer_School_for_High_Performance_Computing_Central]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-13T01:22:04Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}
 Sat 21 Apr 2018 GPC-compute will be decommissioned, GPC-storage available until 9 May 2018 

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-13T01:21:36Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}
 Sat 21 Apr 2018 GPC-compute will be decommissioned, GPC-storage available until 9 May 2018 

 Tue 10 Apr 2018 Niagara is open for general Users

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-04-13T01:21:19Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=https://docs.scinet.utoronto.ca/index.php/Main_Page]][https://docs.scinet.utoronto.ca Niagara]
|-
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}
 Sat 21 Apr 2018 GPC-compute will be decommissioned, GPC-storage available until 9 Ma , 2018 

 Tue 10 Apr 2018 Niagara is open for general Users

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

SciNet User Support Library

2018-04-13T01:18:41Z

Northrup: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==
* April 9, 2018: Niagara commissioned.
* Dec 4, 2017 : scratchtcs decommissioned.
* Nov 28, 2017: The GPC has been reduced from 30,912 to 16,800 cores to make room for the installation of Niagara.
* Sept 29, 2017: The TCS has been decommissioned.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [https://courses.scinet.utoronto.ca SciNet education]
* [https://www.youtube.com/channel/UC42CaO-AAQhwqa8RGzE3daQ SciNet YouTube Channel]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
* [[2016_Ontario_Summer_School_for_High_Performance_Computing_Central]]
* [[2015_Ontario_Summer_School_for_High_Performance_Computing_Central]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

SciNet User Support Library

2018-04-13T01:18:29Z

Northrup: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==
* April 9, 2018: Niagara commissioned for general users.
* Feb 27, 2018: gravity decommissioned.
* Dec 4, 2017 : scratchtcs decommissioned.
* Nov 28, 2017: The GPC has been reduced from 30,912 to 16,800 cores to make room for the installation of Niagara.
* Sept 29, 2017: The TCS has been decommissioned.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [https://courses.scinet.utoronto.ca SciNet education]
* [https://www.youtube.com/channel/UC42CaO-AAQhwqa8RGzE3daQ SciNet YouTube Channel]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
* [[2016_Ontario_Summer_School_for_High_Performance_Computing_Central]]
* [[2015_Ontario_Summer_School_for_High_Performance_Computing_Central]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

Gravity

2018-02-27T18:04:15Z

Northrup:

{{Infobox Computer
|image=[[Image:Ibm_idataplex_dx360_m4.jpg|center|300px|thumb]]
|name=Gravity
|installed=December 2012
|operatingsystem= Linux Centos 6.4
|loginnode= gravity01 (from <tt>login.scinet</tt>)
|nnodes=49 (588 cpu cores, 50176 gpu cores)
|rampernode=32 Gb
|corespernode=12 with 2x M2090 GPUs
|interconnect=QDR Infiniband
|vendorcompilers=nvcc,pgcc,icc,gcc
|queuetype=Torque
}}

'''The Gravity cluster has been decommissioned on February 26, 2018. A new system for large parallel jobs, Niagara, will be replacing the [[GPC_Quickstart|GPC]] and [[TCS_Quickstart|TCS]], and is expected to be in production in early 2018. Contributed systems like [[Sandy]] and [[Gravity]] will also reach their end-of-life in this transition. The aim is to keep at least 50% of the GPC available during the installation of the new system. Users will be informed about further details of the transition as they become available. '''

The Gravity cluster, consists of 49 x86_64 nodes each with two hex core Intel Xeon (Sandybridge) E5-2620 2.0GHz CPUs with 32GB of RAM per node. Each node has two NVIDIA Tesla M2090 GPUs
with CUDA Capability 2.0 (Fermi) each with 512 CUDA Cores and 6 GB of RAM. The nodes are interconnected with 3:1 blocking QDR Infiniband for MPI communications
and disk I/O to the SciNet GPFS filesystems. In total this cluster contains 588 x86_64 cores with 1,568 GB of system RAM and 98 GPUs with 588 GB GPU RAM total.

'''NB -''' gravity is a user-contributed system acquired through a CFI LOF to a specific PI. Policies regarding use by other groups are under development and subject to change at any time.

Note that SciNet has a mailing lists for people interested in GPGPU computing. To receive information on courses, workshop and other GPGPU related events, sign up at https://support.scinet.utoronto.ca/mailman/listinfo/scinet-gpgpu.

== Nodes ==

=== Login ===
First login via ssh with your scinet account at <tt>login.scinet.utoronto.ca</tt>, and from there you can proceed to '''<tt>gravity01</tt>''' which
is the GPU development node.

=== Devel ===

As mentioned '''<tt>gravity01</tt>''' is the head/develop node for interactive use. This node is for compiling, short testing, and submitting
batch jobs to the compute nodes. It is a shared resource so treat it accordingly and use the queue and compute nodes for long are large
computations.

=== ARC Experimental (ARCX) Xeon Phi/ Tesla K20 ===

A separate devel node, '''<tt>arcX</tt>''', with a single Intel Xeon Phi and a NVIDIA Tesla K20 is also available for testing these newer technologies.
For full details see the '''[[phi| Xeon Phi / Tesla K20 ]]''' wiki page.

=== Compute ===

To access the other 48 compute nodes with GPU's you need to use the queue, similar to the standard GPC compute nodes.
Currently the nodes are scheduled by complete node, 12 cores and 2 GPUs, and a maximum walltime of 12 hours.

For an interactive job use
<pre>
qsub -l nodes=1:ppn=12:gpus=2,walltime=12:00:00 -q gravity -I
</pre>

or for a batch job use
<pre>
qsub script.sh
</pre>
where <tt>script.sh</tt> is
<source lang="bash">
#!/bin/bash
# Torque submission script for Gravity
#
#PBS -l nodes=2:ppn=12:gpus=2,walltime=1:00:00
#PBS -N GPUtest
#PBS -q gravity
cd $PBS_O_WORKDIR

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -np 24 ./a.out
</source>

To check running jobs on the gpu nodes only use
<pre>
showq -w class=gravity
</pre>

'''Important note:'''

A bug in the torque scheduler currently sets the environment variable CUDA_VISIBLE_DEVICES to an incorrect value. Loading any one of the cuda modules will correct this, so be to do this in your job script or in your interactive jobs.

== Software ==

The same software installed on the GPC is available on Gravity using the same modules framework.
See [[GPC_Quickstart#Modules_and_Environment_Variables | here]] for full details.

==Programming Frameworks==

Currently there are four programming frameworks to use: NVIDIA's CUDA framework, PGI's CUDA Fortran, PGI's implementation of OpenACC, or OpenCL.

=== NVIDIA toolkit ===

==== Driver Version ====

The current NVIDIA driver version for gravity is 331.67.

==== CUDA ====

The current installed CUDA Toolkits for gravity are 3.2, 4.0, 4.1 (default), 4.2, 5.0, 5.5, and 6.0. A cuda/6.5 module is installed as well, but only for the K20 GPU on the arcX node; that version will not work on the gravity nodes. To use CUDA version 6.0 (recommended for the gravity nodes), just use the following module command

<pre>
module load gcc/4.8.1 cuda/6.0
</pre>
(gcc is a prerequisite of the cuda module; using earlier versions of gcc likely will also work.)



The CUDA driver is installed locally, however the CUDA Toolkits are installed in:

<pre>
/scinet/arc/cuda-$VERSION/
</pre>

The environment variable $SCINET_CUDA_INSTALL is set when a cuda module is loaded and it points to the
install location. This is useful when setting up makefiles and if you use the NVIDIA_SDK
build evironment, modify the NVIDIA_SDK/C/common/common.mk file accordingly.

<pre>
CUDA_INSTALL_PATH = $SCINET_CUDA_INSTALL
</pre>

The Nvidia cuda compiler (which uses gcc/4.4.6 by default for CUDA < 4.1, while cuda/4.2 uses gcc/4.6.1), is called <tt>nvcc</tt>,

You'll have to let the cuda compiler know about the capabilities of the Fermi graphics card by supplying the flag
<pre>-arch=sm_13</pre> or <pre>-arch=sm_20</pre>

==== NVIDIA Toolkit ====

For cuda 5.0 and 5.5, the sdk code samples can be copied from the directory
<pre>
$SCINET_CUDA_INSTALL/samples/
</pre>

NOTE: Not all of the CUDA and OpenCL examples will compile as many require OpenGL graphic libraries not installed on the nodes.

==== OpenCL ====

As of 3.0, OpenCL 1.1 is included in the CUDA Toolkit so loading the CUDA module is all that is required.

===PGI compilers===

As of July 2012, The PGI suite of compilers is installed. These can be accessed by
<pre>$ module load gcc/4.6.1 pgi/12.6</pre>
(if you use the older pgi/12.5, gcc/4.4.6 is a requirement, and is used, for instance, in the CUDA parts of the PGI compilers). These compilers use their own cuda installation, so you do not need to load an additional cuda module. By default, they use a cuda 4.1 installation, but you can request cuda 4.2 as well using the <tt>-Mcuda=4.2</tt> option.

The compilation commands are pgcc, pgcpp and pgfortran for c, c++ and fortran, respectively. As usual, we advice to compiler with optimization using the flags
<pre>
-O4 -fastsse
</pre>
The compilers will then optimize for the specific machine that you are compiling on.

The PGI compilers support OpenMP as well through the compile and link flags
<pre>
-mp
</pre>

==== CUDA Fortran ====

The PGI fortran compiler (<tt>pgfortran</tt>, also <tt>pgf77</tt> and <tt>pgf90</tt>) understands CUDA extensions to fortran.
This compiler will automatically understand these extension for source files with the file extension <tt>.cuf</tt> Otherwise, you have to specify
<pre>
-Mcuda=4.1
</pre>

==== OpenACC ====

OpenACC is a compiler-directive approach to GPGPU programming. The PGI compilers (c, c++ and fortran) have a partial implementation of this open specification. To switch this on, use the options
<pre>-acc -ta=nvidia -Mcuda=4.1</pre>

====More documentation====

Manuals are on the [[Knowledge_Base:_Tutorials_and_Manuals| Tutorials and Manuals]] page.

===Other compilers===

* '''gcc,g++,gfortran''' - GNU compiler (nvcc need to have either gcc-4.4 or gcc-4.6 module loaded to work correctly)
* '''icc,icpc,ifort''' - Intel compiler

===Debuggers===

* '''ddt''' - Allinea's graphical DDT debugger, in the <tt>ddt</tt> module. The most recent version, ddt 4.0 supports cuda 4.0, 4.1, 4.2 and 5.0.
* '''cuda-gdb''' - Nvidia text based gdb variant, part of the cuda module.

Note that to debug both host and cuda device code, you have to give the <pre>-g -G</pre> pair of flags to nvcc.

===MPI===

The GPC MPI packages can be used on this system. See the GPC section on [[ GPC_Quickstart#MPI |MPI ]] for more details.

While these mpi packages should work with the PGI compilers as well, this has not been tested and standard wrappers like mpif90 may not work.

Alternatively, for mpi compilations with the PGI compilers, you can load the mpich1 mpi implementation with <pre>module load mpich1/pgi</pre>after which you can use the option <pre>-Mmpi</pre> or the wrapper scripts <tt>mpicc, mpiCC and mpif90</tt>, as well as <tt>mpirun</tt>.

=== Driver Version ===

The current NVIDIA driver version installed is 331.67

== Documentation ==
* CUDA
** [http://lmgtfy.com/?q=cuda google "CUDA"]

* OpenCL
** see above

== User Codes ==

Please discuss and put any relevant information/problems/best practices you have encountered when using/developing for CUDA and/or OpenCL

Sandy

2018-02-27T18:03:45Z

Northrup:

{{Infobox Computer
|image=[[Image:Ibm_idataplex_dx360_m4.jpg|center|300px|thumb]]
|name=Sandy
|installed=February 2013
|operatingsystem= Linux Centos 6.4
|loginnode= gpc0[1-4] (from <tt>login.scinet</tt>)
|nnodes=44 (704 cores)
|rampernode=64 Gb
|corespernode=16 (32 threads)
|interconnect=QDR Infiniband
|vendorcompilers=icc,gcc
|queuetype=Torque
}}

'''The Sandy cluster will be decommissioned by the end of 2017. A new system for large parallel jobs, Niagara, will be replacing the [[TCS_Quickstart|TCS]], the [[GPC_Quickstart|GPC]] and contributed systems like Sandy, and is expected to be in production in early 2018. The aim is to keep at least 50% of the GPC available during the installation of the new system. Users will be informed about further details of the transition as they become available. '''

The Sandybrdige (Sandy) cluster, consists of 76 x86_64 nodes each with two octal core Intel Xeon (Sandybridge) E5-2650 2.0GHz CPUs with 64GB of RAM per node.
The nodes are interconnected with 2.6:1 blocking QDR Infiniband for MPI communications
and disk I/O to the SciNet GPFS filesystems. In total this cluster contains 1216 x86_64 cores with 4,864 GB of total RAM.

'''NB -''' Sandy is a user-contributed system acquired through a CFI LOF to a specific PI. Policies regarding use by other groups are under development and subject to change at any time.

== Nodes ==

=== Login ===
First login via ssh with your scinet account at <tt>login.scinet.utoronto.ca</tt>, and from there you can proceed to the normal GPC devel nodes '''<tt>gpc0[1-8]</tt>'''.

===Compilers===

The Sandy nodes are fully compatible with all software/modules built for the standard GPC nodes, see [[GPC_Quickstart#Compilers | GPC Quickstart Compilers ]]; however as they a newer architecture they also have added CPU instructions that your program may benefit from. To ensure that you are using these sandy specific optimizations use the following Intel compiler flags with the latest Intel compiler when you compile specifically for the sandy nodes.

<pre>
$ module load intel/14.0.1
</pre>

Optimize your code for the SANDY nodes using the following compiler flags:
<pre>
-O3 -march=corei7-avx
</pre>

*More questions about compiling? See the [[FAQ#Compiling_your_Code|FAQ]].
*NOTE: Code compiled using these option may not be backwards compatible with the regular GPC nodes.

=== Compute ===

To access the sandybridge compute nodes you need to use the queue, similar to the standard GPC compute nodes.
Currently the nodes are scheduled by complete node, 16 cores and a maximum walltime of 48 hours.

For an interactive job use
<pre>
qsub -l nodes=1:ppn=16,walltime=12:00:00 -q sandy -I
</pre>

or for a batch job use
<pre>
qsub script.sh
</pre>
where <tt>script.sh</tt> is
<source lang="bash">
#!/bin/bash
# Torque submission script for Sandy
#
#PBS -l nodes=2:ppn=16,walltime=1:00:00
#PBS -N sandytest
#PBS -q sandy
cd $PBS_O_WORKDIR

# EXECUTION COMMAND; -np = nodes*ppn
mpirun -np 32 ./a.out
</source>

To check running jobs on the sandy nodes only use
<pre>
showq -w class=sandy
</pre>

== Software ==

The same software installed on the GPC is available on Sandy using the same modules framework.
See [[GPC_Quickstart#Modules_and_Environment_Variables | here]] for full details.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-02-27T17:44:07Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 27Feb 2018 09:31:00 EST Gravity decommissioned Tuesday Feb 27th.

 Thu Feb 15 14:39:57 EST 2018 Electrical work is finished, systems back in normal operation.

 Thu Feb 15 8:15:00 EST 2018 Connectivity to the data centre has been restored. Only the P7 system still seems to be unreachable, all other systems are up (within the limitations due to the scheduled partial outage today).

 Thu Feb 14 7:30:00 EST 2018 Connectivity to the data centre failed, so none of the systems can be reached. We are investigating if the connectivity can be restored before today's partial outage is over.

 Sun Feb 11 13:14:44 EST 2018 Scheduled partial system outage February 15th, 8am-5pm, for electrical work. HPSS, atlas and some GPC nodes will be unavailable.

 Sun Feb 11 13:14:44 EST 2018 Gravity GPU Cluster to be decommissioned February 26th.

 Wed Feb 09 10:22:40 EST 2018 HPSS is back in service

 Wed Jan 31 10:16:35 EST 2018 The HPSS core node had a hardware failure overnight. Vendor has been contacted for support and part replacement.

 Mon Jan 29 17:39:03 EST 2018 Maintenance is finished. Most systems will come back online in less than one hour.

Thu Jan 18 21:03:04 EST 2018 '''All systems will be unavailable January 29th 7am thru January 30th for annual scheduled maintenance.'''

Sat Dec 23 13:35:49 EST 2017 All systems have been restored, and jobs are running.

Sat Dec 23 7:00:00 EST 2017 Power outage at the data centre.

SciNet User Support Library

2018-02-27T17:43:21Z

Northrup: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==

* Feb 27, 2017 : gravity decommissioned.

* Dec 4, 2017 : scratchtcs decommissioned.

* Dec 1, 2017, 00:00 am to 06:00 am: The connection to the data center will be down for a scheduled network maintenance. Jobs will continue to run, but login sessions will be terminated at midnight.

* Nov 28, 2017, 12:00 noon: The [[GPC_Quickstart | GPC]] will be reduced from 30,912 to 16,800 cores to make room for the installation of Niagara.

* Sept 29, 2017: The [[TCS_Quickstart | TCS]] was decommissioned on Sept. 29, 2017

* Mar 3: GPC: Version 7.0 of Allinea Forge (DDT Debugger, MAP, Performance Reports) installed as a module.

* Jan 26: New larger (1.8PB) $SCRATCH storage brought online.

* Oct 24: P8: 2 new Power 8 Development Nodes, [[ P8 ]], with 4x Nvidia P100 (Pascal) GPUs, available for users.

* Sept 19: KNL: Intel Knights Landing Development Nodes, [[Knights_Landing | KNL ]], available for users.

* Sept 13: GPC: Version 6.1 of Allinea Forge (DDT Debugger, MAP, Performance Reports) installed as a module.

* Sept 13: GPC: Version 17.0.0 of the Intel Compiler and Tools are installed as modules.

* Aug 20: P8: Power 8 Development Nodes, [[ P8 ]], with 2x Nvidia K80, GPUs available for users.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [https://courses.scinet.utoronto.ca SciNet education]
* [https://www.youtube.com/channel/UC42CaO-AAQhwqa8RGzE3daQ SciNet YouTube Channel]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
* [[2016_Ontario_Summer_School_for_High_Performance_Computing_Central]]
* [[2015_Ontario_Summer_School_for_High_Performance_Computing_Central]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

Oldwiki.scinet.utoronto.ca:System Alerts

2018-02-27T17:42:46Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 27Feb 2018 09:31:00 EST Gravity decommissioned Tuesday Feb 27th.

 Thu Feb 15 14:39:57 EST 2018 Electrical work is finished, systems back in normal operation.

 Thu Feb 15 8:15:00 EST 2018 Connectivity to the data centre has been restored. Only the P7 system still seems to be unreachable, all other systems are up (within the limitations due to the scheduled partial outage today).

 Thu Feb 14 7:30:00 EST 2018 Connectivity to the data centre failed, so none of the systems can be reached. We are investigating if the connectivity can be restored before today's partial outage is over.

 Sun Feb 11 13:14:44 EST 2018 Scheduled partial system outage February 15th, 8am-5pm, for electrical work. HPSS, atlas and some GPC nodes will be unavailable.

 Sun Feb 11 13:14:44 EST 2018 Gravity GPU Cluster to be decommissioned February 26th.

 Wed Feb 09 10:22:40 EST 2018 HPSS is back in service

 Wed Jan 31 10:16:35 EST 2018 The HPSS core node had a hardware failure overnight. Vendor has been contacted for support and part replacement.

 Mon Jan 29 17:39:03 EST 2018 Maintenance is finished. Most systems will come back online in less than one hour.

Thu Jan 18 21:03:04 EST 2018 '''All systems will be unavailable January 29th 7am thru January 30th for annual scheduled maintenance.'''

Sat Dec 23 13:35:49 EST 2017 All systems have been restored, and jobs are running.

Sat Dec 23 7:00:00 EST 2017 Power outage at the data centre.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-02-27T03:33:09Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 26 Feb 2018 09:31:00 EST Gravity will be decommissioned Tuesday Feb 27th.

 Thu Feb 15 14:39:57 EST 2018 Electrical work is finished, systems back in normal operation.

 Thu Feb 15 8:15:00 EST 2018 Connectivity to the data centre has been restored. Only the P7 system still seems to be unreachable, all other systems are up (within the limitations due to the scheduled partial outage today).

 Thu Feb 14 7:30:00 EST 2018 Connectivity to the data centre failed, so none of the systems can be reached. We are investigating if the connectivity can be restored before today's partial outage is over.

 Sun Feb 11 13:14:44 EST 2018 Scheduled partial system outage February 15th, 8am-5pm, for electrical work. HPSS, atlas and some GPC nodes will be unavailable.

 Sun Feb 11 13:14:44 EST 2018 Gravity GPU Cluster to be decommissioned February 26th.

 Wed Feb 09 10:22:40 EST 2018 HPSS is back in service

 Wed Jan 31 10:16:35 EST 2018 The HPSS core node had a hardware failure overnight. Vendor has been contacted for support and part replacement.

 Mon Jan 29 17:39:03 EST 2018 Maintenance is finished. Most systems will come back online in less than one hour.

Thu Jan 18 21:03:04 EST 2018 '''All systems will be unavailable January 29th 7am thru January 30th for annual scheduled maintenance.'''

Sat Dec 23 13:35:49 EST 2017 All systems have been restored, and jobs are running.

Sat Dec 23 7:00:00 EST 2017 Power outage at the data centre.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-02-26T14:32:21Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Mon 26 Feb 2018 09:31:00 EST Gravity will be decommissioned Tuesday Feb 27th. 

 Thu Feb 15 14:39:57 EST 2018 Electrical work is finished, systems back in normal operation.

 Thu Feb 15 8:15:00 EST 2018 Connectivity to the data centre has been restored. Only the P7 system still seems to be unreachable, all other systems are up (within the limitations due to the scheduled partial outage today).

 Thu Feb 14 7:30:00 EST 2018 Connectivity to the data centre failed, so none of the systems can be reached. We are investigating if the connectivity can be restored before today's partial outage is over.

 Sun Feb 11 13:14:44 EST 2018 Scheduled partial system outage February 15th, 8am-5pm, for electrical work. HPSS, atlas and some GPC nodes will be unavailable.

 Sun Feb 11 13:14:44 EST 2018 Gravity GPU Cluster to be decommissioned February 26th.

 Wed Feb 09 10:22:40 EST 2018 HPSS is back in service

 Wed Jan 31 10:16:35 EST 2018 The HPSS core node had a hardware failure overnight. Vendor has been contacted for support and part replacement.

 Mon Jan 29 17:39:03 EST 2018 Maintenance is finished. Most systems will come back online in less than one hour.

Thu Jan 18 21:03:04 EST 2018 '''All systems will be unavailable January 29th 7am thru January 30th for annual scheduled maintenance.'''

Sat Dec 23 13:35:49 EST 2017 All systems have been restored, and jobs are running.

Sat Dec 23 7:00:00 EST 2017 Power outage at the data centre.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-02-15T19:41:33Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Thu Feb 15 14:39:57 EST 2018 Electrical work is finished, systems back in normal operation.

 Thu Feb 15 8:15:00 EST 2018 Connectivity to the data centre has been restored. Only the P7 system still seems to be unreachable, all other systems are up (within the limitations due to the scheduled partial outage today).

 Thu Feb 14 7:30:00 EST 2018 Connectivity to the data centre failed, so none of the systems can be reached. We are investigating if the connectivity can be restored before today's partial outage is over.

 Sun Feb 11 13:14:44 EST 2018 Scheduled partial system outage February 15th, 8am-5pm, for electrical work. HPSS, atlas and some GPC nodes will be unavailable.

 Sun Feb 11 13:14:44 EST 2018 Gravity GPU Cluster to be decommissioned February 26th.

 Wed Feb 09 10:22:40 EST 2018 HPSS is back in service

 Wed Jan 31 10:16:35 EST 2018 The HPSS core node had a hardware failure overnight. Vendor has been contacted for support and part replacement.

 Mon Jan 29 17:39:03 EST 2018 Maintenance is finished. Most systems will come back online in less than one hour.

Thu Jan 18 21:03:04 EST 2018 '''All systems will be unavailable January 29th 7am thru January 30th for annual scheduled maintenance.'''

Sat Dec 23 13:35:49 EST 2017 All systems have been restored, and jobs are running.

Sat Dec 23 7:00:00 EST 2017 Power outage at the data centre.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-02-15T14:09:05Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:down.png|down|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Thu Feb 15 8:15:00 EST 2018 Connectivity to the data centre has been restored. Only the P7 system still seems to be unreachable, all other systems are up (within the limitations due to the scheduled partial outage today).

 Thu Feb 14 7:30:00 EST 2018 Connectivity to the data centre failed, so none of the systems can be reached. We are investigating if the connectivity can be restored before today's partial outage is over.

 Sun Feb 11 13:14:44 EST 2018 Partial system outage February 15th, 8am-5pm, for electrical work. HPSS, atlas and some GPC nodes will be unavailable.

 Sun Feb 11 13:14:44 EST 2018 Gravity GPU Cluster to be decommissioned February 26th.

 Wed Feb 09 10:22:40 EST 2018 HPSS is back in service

 Wed Jan 31 10:16:35 EST 2018 The HPSS core node had a hardware failure overnight. Vendor has been contacted for support and part replacement.

 Mon Jan 29 17:39:03 EST 2018 Maintenance is finished. Most systems will come back online in less than one hour.

Thu Jan 18 21:03:04 EST 2018 '''All systems will be unavailable January 29th 7am thru January 30th for annual scheduled maintenance.'''

Sat Dec 23 13:35:49 EST 2017 All systems have been restored, and jobs are running.

Sat Dec 23 7:00:00 EST 2017 Power outage at the data centre.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-02-15T13:44:47Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:down.png|down|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Thu Feb 14 8:15:00 EST 2018 Connectivity to the data centre has been restored. Only the P7 system still seems to be unreachable, all other systems are up (within the limitations due to the scheduled partial outage today).

 Thu Feb 14 7:30:00 EST 2018 Connectivity to the data centre failed, so none of the systems can be reached. We are investigating if the connectivity can be restored before today's partial outage is over.

 Sun Feb 11 13:14:44 EST 2018 Partial system outage February 15th, 8am-5pm, for electrical work. HPSS, atlas and some GPC nodes will be unavailable.

 Sun Feb 11 13:14:44 EST 2018 Gravity GPU Cluster to be decommissioned February 26th.

 Wed Feb 09 10:22:40 EST 2018 HPSS is back in service

 Wed Jan 31 10:16:35 EST 2018 The HPSS core node had a hardware failure overnight. Vendor has been contacted for support and part replacement.

 Mon Jan 29 17:39:03 EST 2018 Maintenance is finished. Most systems will come back online in less than one hour.

Thu Jan 18 21:03:04 EST 2018 '''All systems will be unavailable January 29th 7am thru January 30th for annual scheduled maintenance.'''

Sat Dec 23 13:35:49 EST 2017 All systems have been restored, and jobs are running.

Sat Dec 23 7:00:00 EST 2017 Power outage at the data centre.

Oldwiki.scinet.utoronto.ca:System Alerts

2018-02-11T18:18:19Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|
|-
|[[File:up.png|up|]]File System
|[[File:up.png|up|]]Scheduler
|[[File:up.png|up|]]External Network
|
|}

 Sun Feb 11 13:14:44 EST 2018 Partial system outage February 15th, 8am-5pm, for electrical work. HPSS, atlas and some GPC nodes will be unavailable.

 Sun Feb 11 13:14:44 EST 2018 Gravity GPU Cluster to be decommissioned February 26th.

 Wed Feb 09 10:22:40 EST 2018 HPSS is back in service

 Wed Jan 31 10:16:35 EST 2018 The HPSS core node had a hardware failure overnight. Vendor has been contacted for support and part replacement.

 Mon Jan 29 17:39:03 EST 2018 Maintenance is finished. Most systems will come back online in less than one hour.

Thu Jan 18 21:03:04 EST 2018 '''All systems will be unavailable January 29th 7am thru January 30th for annual scheduled maintenance.'''

Sat Dec 23 13:35:49 EST 2017 All systems have been restored, and jobs are running.

Sat Dec 23 7:00:00 EST 2017 Power outage at the data centre.

P8

2018-02-06T01:17:57Z

Northrup: /* IBM Compilers */

{{Infobox Computer
|image=[[Image:P8_s822.jpg|center|300px|thumb]]
|name=P8
|installed=June 2016
|operatingsystem= Linux RHEL 7.2 le / Ubuntu 16.04 le
|loginnode= p8t0[1-2] / p8t0[3-4]
|nnodes= 2x Power8 with 2x NVIDIA K80, 2x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 8core (16 physical, 128 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== Specifications==

The P8 Test System consists of of 4 IBM Power 822LC Servers each with 2x8core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 16 physical cores to support up to 128 threads. 2 nodes have two NVIDIA Tesla K80 GPUs with CUDA Capability 3.7 (Kepler), consisting of 2xGK210 GPUs each with 12 GB of RAM connected using PCI-E, and 2 others have 4x NVIDIA Tesla P100 GPUs each wit h 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Compile/Devel/Test ==

First login via ssh with your scinet account at '''<tt>login.scinet.utoronto.ca</tt>''', and from there you can ssh to <tt>p8t01</tt> or <tt>p8t02</tt> for the K80 GPUs and to <tt>p8t03</tt> or <tt>p8t04</tt> for the Pascal GPUs.

== Software for ==

==== GNU Compilers ====

To load the newer advance toolchain version use:

For '''<tt>p8t0[1-2]</tt>'''
<pre>
module load gcc/5.3.1
</pre>

For '''<tt>p8t0[3-4]</tt>'''
<pre>
module load gcc/6.2.1
</pre>

==== IBM Compilers ====

To load the native IBM xlc/xlc++ compilers

For '''<tt>p8t0[1-2]</tt>'''
<pre>
module load xlc/13.1.4
module load xlf/13.1.4
</pre>

For '''<tt>p8t0[3-4]</tt>'''
<pre>
module load xlc/13.1.5_b2
module load xlf/13.1.5_b2
</pre>

==== Driver Version ====

The current NVIDIA driver version is 361.93

==== CUDA ====

The current installed CUDA Tookit is 8.0

<pre>
module load cuda/8.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
</pre>

==== OpenMPI ====

Currently OpenMPI has been setup on the four nodes connected over QDR Infiniband.

For '''<tt>p8t0[1-2]</tt>'''
<pre>
$ module load openmpi/1.10.3-gcc-5.3.1
$ module load openmpi/1.10.3-XL-13_15.1.4
</pre>

For '''<tt>p8t0[3-4]</tt>'''
<pre>
$ module load openmpi/1.10.3-gcc-6.2.1
$ module load openmpi/1.10.3-XL-13_15.1.5
</pre>

==== PE ====

IBM's Parallel Environment (PE), is available for use with XL compilers using the following

<pre>
$ module pe/xl.perf
</pre>

<pre>
mpiexec -n 4 ./a.out
</pre>

documentation is [http://publib.boulder.ibm.com/epubs/pdf/c2372832.pdf here]

File:Intro Niagara.pdf

2018-01-25T14:17:27Z

Northrup:

File:Snug techtalk Niagara.pdf

2018-01-25T14:15:18Z

Northrup:

Oldwiki.scinet.utoronto.ca:System Alerts

2018-01-19T02:05:40Z

Northrup: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2018-01-19T02:04:53Z

Northrup:

Oldwiki.scinet.utoronto.ca:System Alerts

2018-01-19T02:02:18Z

Northrup: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2018-01-19T01:59:44Z

Northrup: /* System Status */

SOSCIP GPU

2018-01-04T13:34:13Z

Northrup: /* LINKS */

{{Infobox Computer
|image=[[Image:S882lc.png|center|300px|thumb]]
|name=SOSCIP GPU
|installed=September 2017
|operatingsystem= Ubuntu 16.04 le
|loginnode= sgc01
|nnodes= 14x Power 8 with 4x NVIDIA P100
|rampernode=512 GB
|corespernode= 2 x 10core (20 physical, 160 SMT)
|interconnect=Infiniband EDR
|vendorcompilers=xlc/xlf, nvcc
}}

== SOSCIP ==

The SOSCIP GPU Cluster is a Southern Ontario Smart Computing Innovation Platform ([http://soscip.org/ SOSCIP]) resource located at theUniversity of Toronto's SciNet HPC facility. The SOSCIP multi-university/industry consortium is funded by the Ontario Government and the Federal Economic Development Agency for Southern Ontario [http://www.research.utoronto.ca/about/our-research-partners/soscip/].

== Support Email ==

Please use [mailto:soscip-support@scinet.utoronto.ca <soscip-support@scinet.utoronto.ca>] for SOSCIP GPU specific inquiries.

== Specifications==

The SOSCIP GPU Cluster consists of of 14 IBM Power 822LC "Minsky" Servers each with 2x10core 3.25GHz Power8 CPUs and 512GB Ram. Similar to Power 7, the Power 8 utilizes Simultaneous MultiThreading (SMT), but extends the design to 8 threads per core allowing the 20 physical cores to support up to 160 threads. Each node has 4x NVIDIA Tesla P100 GPUs each with 16GB of RAM with CUDA Capability 6.0 (Pascal) connected using NVlink.

== Compile/Devel/Test ==

Access is provided through the BGQ login node, '''<tt> bgqdev.scinet.utoronto.ca </tt>''' using ssh, and from there you can proceed to the GPU development node '''<tt>sgc01-ib0</tt>'''.

== Filesystem ==

The filesystem is shared with the BGQ system. See [https://wiki.scinet.utoronto.ca/wiki/index.php/BGQ#Filesystem here ] for details.

== Job Submission ==

The SOSCIP GPU cluster uses [https://slurm.schedmd.com/ SLURM ] as a job scheduler and jobs are scheduled by node, ie 20 cores and 4 GPUs each. Jobs are submitted from the development node '''<tt>sgc01</tt>'''. The maximum walltime per job is 12 hours (except in the 'long' queue, see below) with up to 8 nodes.

<pre>
$ sbatch myjob.script
</pre>

Where myjob.script is

<pre>
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

hostname
nvidia-smi
</pre>

You can queury job information using

<pre>
squeue
</pre>

To cancel a job use

<pre>
scancel $JOBID
</pre>

=== Longer jobs ===

If your job takes more than 12 hours, the sbatch command will not let you submit your job. There is, however, a way to have jobs up to 24 hours long, by specifying "-p long" as an option (i.e., add <tt>#SBATCH -p long</tt> to your job script). The priority of such jobs may be throttled in the future if we see that the 'long' queue is having a negative efffect on turnover time in the queue.

=== Interactive ===

For an interactive session use

<pre>
salloc --gres=gpu:4
</pre>

=== Automatic Re-submission and Job Dependencies ===

Commonly you may have a job that you know will take longer to run than what is permissible in the queue. As long as your program contains checkpoint or restart capability, you can have one job automatically submit the next. In the following example it is assumed that the program finishes before the time limit requested and then resubmits itself by logging into the development nodes. Job dependencies and a maximum number of job re-submissions are used to ensure sequential operation.

<pre>
#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=20 # MPI tasks (needed for srun)
#SBATCH --time=00:10:00 # H:M:S
#SBATCH --gres=gpu:4 # Ask for 4 GPUs per node

cd $SLURM_SUBMIT_DIR

: ${job_number:="1"} # set job_nubmer to 1 if it is undefined
job_number_max=3

echo "hi from ${SLURM_JOB_ID}"

#RUN JOB HERE

# SUBMIT NEXT JOB
if [[ ${job_number} -lt ${job_number_max} ]]
then
(( job_number++ ))
next_jobid=$(ssh sgc01-ib0 "cd $SLURM_SUBMIT_DIR; /opt/slurm/bin/sbatch --export=job_number=${job_number} -d afterok:${SLURM_JOB_ID} thisscript.sh | awk '{print $4}'")
echo "submitted ${next_jobid}"
fi

sleep 15

echo "${SLURM_JOB_ID} done"

</pre>

== Software Installed ==

=== IBM PowerAI ===

The PowerAI platform contains popular open machine learning frameworks such as '''Caffe, TensorFlow, and Torch'''. Run the <tt>module avail</tt> command for a complete listing. More information is available at this link: https://developer.ibm.com/linuxonpower/deep-learning-powerai/releases/. Release 4.0 is currently installed.

==== GNU Compilers ====

More recent versions of the GNU Compiler Collection (C/C++/Fortran) are provided in the IBM Advanced Toolchain with enhancements for the POWER8 CPU. To load the newer advance toolchain version use:

Advanced Toolchain V10.0
<pre>
module load gcc/6.3.1
</pre>

Advanced Toolchain V11.0
<pre>
module load gcc/7.2.1
</pre>

More information about the IBM Advanced Toolchain can be found here: [https://developer.ibm.com/linuxonpower/advance-toolchain/ https://developer.ibm.com/linuxonpower/advance-toolchain/]

==== IBM XL Compilers ====

To load the native IBM xlc/xlc++ and xlf (Fortran) compilers, run

<pre>
module load xlc/13.1.5
module load xlf/15.1.5
</pre>

IBM XL Compilers are enabled for use with NVIDIA GPUs, including support for OpenMP 4.5 GPU offloading and integration with NVIDIA's nvcc command to compile host-side code for the POWER8 CPU.

Information about the IBM XL Compilers can be found at the following links:

[https://www.ibm.com/support/knowledgecenter/SSXVZZ_13.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL C/C++]

[https://www.ibm.com/support/knowledgecenter/SSAT4T_15.1.5/com.ibm.compilers.linux.doc/welcome.html IBM XL Fortran]

==== NVIDIA GPU Driver Version ====

The current NVIDIA driver version is 384.66

==== CUDA ====

The current installed CUDA Tookits is are version 8.0 and version 9.0

<pre>
module load cuda/8.0
</pre>

or

<pre>
module load cuda/9.0
</pre>

The CUDA driver is installed locally, however the CUDA Toolkit is installed in:

<pre>
/usr/local/cuda-8.0
/usr/local/cuda-9.0
</pre>

Note that the <tt>/usr/local/cuda</tt> directory is linked to the <tt>/usr/local/cuda-9.0</tt> directory.

Documentation and API reference information for the CUDA Toolkit can be found here: [http://docs.nvidia.com/cuda/index.html http://docs.nvidia.com/cuda/index.html]

==== OpenMPI ====

Currently OpenMPI has been setup on the 14 nodes connected over EDR Infiniband.

<pre>
$ module load openmpi/2.1.1-gcc-5.4.0
$ module load openmpi/2.1.1-XL-13_15.1.5
</pre>

== Other Software ==

Other software packages can be installed onto the SOSCIP GPU Platform. It is best to try installing new software in your own home directory, which will give you control of the software (e.g. exact version, configuration, installing sub-packages, etc.).

In the following subsections are instructions for installing several common software packages.

=== Anaconda (Python) ===

Anaconda is a popular distribution of the Python programming language. It contains several common Python libraries such as SciPy and NumPy as pre-built packages, which eases installation.

Anaconda can be downloaded from here: [https://www.anaconda.com/download/#linux https://www.anaconda.com/download/#linux]

NOTE: Be sure to download the '''Power8''' installer.

TIP: If you plan to use Tensorflow within Anaconda, download the Python 2.7 version of Anaconda

=== Keras ===

Keras ([https://keras.io/ https://keras.io/]) is a popular high-level deep learning software development framework. It runs on top of other deep-learning frameworks such as TensorFlow.

The easiest way to install Keras is to install Anaconda first, then install Keras by using using the pip command.

Keras uses TensorFlow underneath to run neural network models. Before running code using Keras, be sure to load the PowerAI TensorFlow module and the cuda module.

=== PyTorch ===

PyTorch is the Python implementation of the Torch framework for deep learning.

It is suggested that you use PyTorch within Anaconda.

There is currently no build of PyTorch for POWER8-based systems. You will need to compile it from source.

Obtain the source code from here: [http://pytorch.org/ http://pytorch.org/]

Before building PyTorch, make sure to load cuda by running

<pre>
module load cuda/8.0
</pre>

NOTE: Do not have the gcc modules loaded when building PyTorch. Use the default version of gcc (currently v5.4.0) included with the operating system. Build will fail with later versions of gcc.

== LINKS ==

[https://www.olcf.ornl.gov/kb_articles/summitdev-quickstart/#System_Overview Summit Dev System at ORNL]

SOSCIP GPU

2018-01-04T13:34:02Z

Northrup:

SciNet User Support Library

2017-12-04T15:21:06Z

Northrup: /* System News */

__NOTOC__
{| style="border-spacing: 8px;"
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
{{SciNetWiki:System_Alerts}}
| valign="top" style="cellpadding:1em; padding:1em; border:3px solid #0645ad; background-color:#f6f6f6; border-radius:7px"|
==System News==

* Dec 4, 2017 : scratchtcs decommissioned.

* Dec 1, 2017, 00:00 am to 06:00 am: The connection to the data center will be down for a scheduled network maintenance. Jobs will continue to run, but login sessions will be terminated at midnight.

* Nov 28, 2017, 12:00 noon: The [[GPC_Quickstart | GPC]] will be reduced from 30,912 to 16,800 cores to make room for the installation of Niagara.

* Sept 29, 2017: The [[TCS_Quickstart | TCS]] was decommissioned on Sept. 29, 2017

* Mar 3: GPC: Version 7.0 of Allinea Forge (DDT Debugger, MAP, Performance Reports) installed as a module.

* Jan 26: New larger (1.8PB) $SCRATCH storage brought online.

* Oct 24: P8: 2 new Power 8 Development Nodes, [[ P8 ]], with 4x Nvidia P100 (Pascal) GPUs, available for users.

* Sept 19: KNL: Intel Knights Landing Development Nodes, [[Knights_Landing | KNL ]], available for users.

* Sept 13: GPC: Version 6.1 of Allinea Forge (DDT Debugger, MAP, Performance Reports) installed as a module.

* Sept 13: GPC: Version 17.0.0 of the Intel Compiler and Tools are installed as modules.

* Aug 20: P8: Power 8 Development Nodes, [[ P8 ]], with 2x Nvidia K80, GPUs available for users.

([[Previous System News]])
|-
| valign="top" width="50%" style="padding:1em; border:1px solid #aaaaaa; background-color:#f6e8e8; border-radius:7px" |

== QuickStart Guides ==
* [[SciNet Command Line Utilities]]
* [[Media:SciNet_Tutorial.pdf|SciNet User Tutorial]]
* [[GPC_Quickstart|GPC: General Purpose Cluster]]
* [[Sandy| Sandy: Intel Sandybridge nodes ]]
* [[Gravity| Gravity: GPU nodes]]
* [[P7_Linux_Cluster|P7: Power 7 Linux cluster]]
* [[BGQ|BGQ: BlueGene/Q clusters]]
* [[Software_and_Libraries|Software and libraries]]
* [[Data_Management|Data management]]
* [[FAQ | FAQ (frequently asked questions)]]
* [[Acknowledging_SciNet | Acknowledging SciNet]]
* [https://courses.scinet.utoronto.ca SciNet education]
* [https://www.youtube.com/channel/UC42CaO-AAQhwqa8RGzE3daQ SciNet YouTube Channel]
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8f6e8; border-radius:7px" |

== Tutorials and Manuals ==
* [[Knowledge_Base:_Tutorials_and_Manuals#SciNet_Basics|SciNet basics]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Linux|Linux]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Batch_job_management|Batch job management]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Programming|Programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Parallel_Programming|Parallel programming]]
* [[Knowledge_Base:_Tutorials_and_Manuals#GPU_Computing|GPU computing]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Performance_Tuning|Performance tuning]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Debugging|Debugging]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Math_libraries_.28BLAS.2C_LAPACK.2C_FFT.29|Math libraries (BLAS, LAPACK, FFT)]]
* [[Knowledge_Base:_Tutorials_and_Manuals#I/O|I/O and databases]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Visualization|Visualization]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Applications|Applications]]
* [[Knowledge_Base:_Tutorials_and_Manuals#Manuals|Manuals (compilers etc)]]
* [[2016_Ontario_Summer_School_for_High_Performance_Computing_Central]]
* [[2015_Ontario_Summer_School_for_High_Performance_Computing_Central]]
|-
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#e8e8f6; border-radius:7px" |

== What's New On The Wiki ==
<div style='text-align:left;'>

* Dec 2014: Updated [[GPC Quickstart]] with info on email notifications from the scheduler.

* Dec 2014: [[Hdf5]] compilation page updated.

* Dec 2014: [[Research Computing with Python 2014]] lectures 5-8

* Nov 2014: "Modern CUDA Features" TechTalk slides in [[SciNet TechTalks and Seminars]].

* Nov 2014: [[Research Computing with Python 2014]] lectures 1-4

* Sept 2014: [[User_Tips#Reducing_virtual_memory_consumption_for_multithreaded_programs| Tip on reducing virtual memory consumption for multithreaded programs]]

* Sept 2014: Improved information on the [[Python]] versions installed on the GPC, and which modules are included in each version.

* Sept 2014: Description on using job arrays on the GPC on the [[Scheduler]] page.

* Sept 2014: [[Intro to Tkinter|Tkinter instructions]], [[Media:Tkinter.pdf|slides]] and [[Media:Tkinter_code.tgz|code]] for the TkInter workshop held in September.

* Sept 2014: Instructions on using [[Hadoop for HPCers|Hadoop]] (for the Hadoop workshop held in September).

</div>
Previous new stuff can be found in the [[What's new archive]].
| valign="top" style="padding:1em; border:1px solid #aaaaaa; background-color:#f4f4fa; border-radius:7px" |
<div style='text-align:left;'>
{{SciNetWiki:Community_Portal}}
</div>
|}

{{#Widget:Twitter|shell.background=#9fb1c2|tweets.links=#4775c1|tweets.color=#000000|tweets.background=#ffffff|user=SciNetHPC}}

<!-- [[Old Main Page]] --!>

SOSCIP GPU

2017-11-23T16:08:19Z

Northrup: /* GNU Compilers */

SOSCIP GPU

2017-11-23T16:07:47Z

Northrup: /* GNU Compilers */

SOSCIP GPU

2017-11-23T16:07:15Z

Northrup: /* GNU Compilers */

Oldwiki.scinet.utoronto.ca:System Alerts

2017-11-13T04:33:36Z

Northrup: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|-
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|[[File:up.png|up|]]File System
|-
|[[File:up.png|up|link=SOSCIP_GPU]][[SOSCIP_GPU|SGC]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|[[File:up.png|up|]]External Network

|[[File:up.png|up|]]Login Nodes
|}

 Sun Nov 12 23:33:04 EST 2017 File system issues cleared. System should be back to normal.

 Sat Nov 11, 13:43 EDT 2017 File system issues. You may experience problems logging in. We are investigating.

 Sat Nov 11, 12:19 EDT 2017 If you are experiencing issues when logging in, try ssh to login-ha.scinet.utoronto.ca.

 Thu Oct 19, 18:30 EDT 2017 Connectivity to the datacentre has been restored.

 Thu Oct 19, 17:00 EDT 2017 [http://www.systemstatus.utoronto.ca/incidents/zlj7rd5z2mg4 It appears that the optical fibre connection to the datacentre has been cut, some 3 km away from the University. Bell has been contacted.]

 Thu Oct 19, 14:30 EDT 2017 [http://www.systemstatus.utoronto.ca/incidents/zlj7rd5z2mg4 Network issues to the data centre. We are investigating.]

SOSCIP GPU

2017-11-10T06:30:28Z

Northrup: /* = Automatic Re-submission and Job Dependencies */

SOSCIP GPU

2017-11-10T06:30:04Z

Northrup: /* Job Submission */

Oldwiki.scinet.utoronto.ca:System Alerts

2017-10-30T14:14:20Z

Northrup: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2017-10-30T13:37:32Z

Northrup: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2017-10-28T17:52:32Z

Northrup: /* System Status */

SOSCIP GPU

2017-10-10T18:09:15Z

Northrup: /* OpenAI */