oldwiki.scinet.utoronto.ca - User contributions [en-gb]

Template:SciNetWiki:System Alerts

2021-03-17T23:16:09Z

Chen: Created page with "== System Status== <!-- Notes for updating the system status: - When removing system status entries, please archive them to: http://wiki.scinethpc.ca/wiki/index.p..."

== System Status==


System status can now be found at [https://docs.scinet.utoronto.ca docs.scinet.utoronto.ca]

 Mon 23 Apr 2018 GPC-compute is decommissioned, GPC-storage available until 30 May 2018

 Thu 18 Apr 2018 Niagara system will undergo an upgrade to its Infiniband network between 9am and 12pm, should be transparent to users, however there is a chance of network interruption.

 Fri 13 Apr 2018 HPSS system will be down for a few hours on Mon, Apr/16, 9AM, for hardware upgrades, in preparation for the eventual move to the Niagara side.

 Tue 10 Apr 2018 Niagara is open to users.

 Wed 4 Apr 2018 We are very close to the production launch of Niagara, the new system installed at SciNet.
While the RAC allocation year officially starts today, April 4/18, the Niagara system is still undergoing some final tuning and software updates, so the plan is to officially open it to users on next week.

All active GPC users will have their accounts, $HOME, and $PROJECT, transferred to the new
Niagara system. Those of you who are new to SciNet, but got RAC allocations on Niagara,
will have your accounts created and ready for you to login.

We are planning an extended [https://support.scinet.utoronto.ca/education/go.php/370/index.php Intro to SciNet/Niagara session], available in person at our office, and webcast on Vidyo and possibly other means, on Wednesday April 11 at noon EST.

Template:SciNetWiki:Community Portal

2021-03-17T23:09:17Z

Chen: Created page with "==User-Supplied Content== '''Share your expertise with the SciNet Community!''' * Running specific codes on SciNet machines * User_Tips | Tips for performa..."

==User-Supplied Content==
'''Share your expertise with the SciNet Community!'''
* [[User_Codes | Running specific codes on SciNet machines]]
* [[User_Tips | Tips for performance or convenience on SciNet machines]]
* [[User_Serial | Running serial batch jobs (including GNU parallel)]]
* [[User_Ramdisk | Using the ramdisk on SciNet's GPC]]
* [[Perl | User-space perl modules]]
* [[HPSS-by-pomes|Packing up large data sets and putting them on HPSS]]
* [[User_Remote | Remote batch]]

Oldwiki.scinet.utoronto.ca:System Alerts

2017-08-18T20:30:17Z

Chen: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=BGQ]][[BGQ]]
|[[File:up.png|up|]]File System
|-
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=P8]][[P8]]
|[[File:up.png|up|link=Knights Landing]][[Knights Landing|KNL]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|}

 Fri Aug 18 13:20:56 EDT 2017 We need to have an emergency shutdown of all compute systems to fix a cooling issue that has arisen. Should be back up this afternoon. We will try and keep the login nodes and stroage online, however all the compute nodes will need to be shutdown.

Sat Aug 5 16:48:44 EDT 2017 The switch is fixed. Scinet0[2-4] and datamovers are back online.

Sat Aug 5 00:56:31 EDT 2017 Most of GPC will be accessible soon. Lost a switch, scinet0[2-4] and datamovers will be down until it's fixed. Scinet01 may be login using its IP address; "ssh 142.150.188.51".

Fri Aug 4 17:45:10 EDT 2017 The chiller went down again, causing a full shutdown of all systems. We don't expect them back tonight, as the storm continues and power outages continue with it.

Fri Aug 4 17:11:08 EDT 2017 A power glitch took down all the compute nodes, including GPC, TCS, BGQ. The filesystems are up, except for reserved1 and scratchtcs. Systems are being restored.

Tue Jun 13 11:24:15 EDT 2017 HPSS is back on service.

Sat Jun 10 21:38:37 EDT 2017 The robot arm on the HPSS library developed problems. HPSS is out of service until further notice.

Tue May 23 13:40:56 EDT 2017 HPSS is back on service.

Mon May 22 07:37:27 EDT 2017 Overnight the robot arm on the library developed problems. We may not be able to have support come in and have it fixed until Tuesday. In the meantime HPSS is out of service

Thu May 18 13:00:00 EDT 2017 File system seem resolved. Please check your jobs and resubmit if they had issues.

Thu May 18 12:00:00 EDT 2017 File system issues and some jobs may have died. Investigating.

Oldwiki.scinet.utoronto.ca:System Alerts

2016-09-25T16:51:29Z

Chen: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up]]File System
|-
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|[[File:up.png|down|link=BGQ]][[BGQ]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|}

Sun Sep 25 12:37:12 EDT 2016 Problems resolved. Systems have started coming online. Check the status "lights" above.

Sun Sep 25 10:16:37 EDT 2016 Power outage tripped main breaker and other circuits. Power has been restored to site but there may be an issue with cooling system power that needs to be resolved before any compute systems can be restarted

Sun Sep 25 09:28:15 EDT 2016 Staff enroute to site. After assessing situation will give ETA for recovery.

Sun Sep 25 08:46 EDT 2016 Power outage at datacentre.

Thu Sep 8 10:46:22 EDT 2016 Until further notice HPSS jobs are being manually throttled to ensure the cache does not get overwhelmed. We have new tape drives being added to the system soon, and that will help alleviate the problem when we return to normal operations.

Fri Sep 2 17:31:44 EDT 2016 The HPSS disk-based cache is full, and currently being drained to tape. Until enough material has been migrated and purged the system will remain on hold.

Sun Aug 28 16:10:56 EDT BGQ racks were shutoff due to environment issue, system being restarted. Please resubmit your jobs.

Sun 28 Aug 2016 10:19:17 EDT Turning systems on. Expect to be up in a couple of hours.

Sun Aug 28 09:16:02 EDT 2016 Datacentre was shutdown due to facility power failure.

Tue 16 Aug 2016 21:55:20 Cooling restored, filesystem up and OK, bringing up clusters. GPC should be available to users by 11PM (perhaps as early as 1030)

Tue 16 Aug 2016 20:46:37 Water service has been restored to building. Restarting cooling system.

Tue 16 Aug 2016 19:52:32 SciNet-related maintenance and modifications have been completed successfully. Work on the building water valve is expected to be done on time (by 9PM). Once water service is restored we need to restore cooling, power-up filesystems and then restart the clusters. Unlikely that any systems are available to users before 11PM and it will take longer to get everything online. Check here for updates

Tue 16 Aug 2016 07:07:32 Shutdown has started

 Scheduled full-day maintenance shutdown begins:

7AM, Tuesday, 16 Aug 

Several projects (adding new 208V circuits for storage, cooling tower maintenance etc) are being carried out on same day as the landlord needs to shutdown the main building water supply (and therefore our cooling system as well) for repairs.

Expect to start bringing systems up about 10PM depending on when the water work is done. Check here for further updates during the day

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-28T23:17:27Z

Chen: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up]]File System
|-
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|up|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|[[File:up75.png|down|link=BGQ]][[BGQ]]
|[[File:up.png|up|link=HPSS]][[HPSS]]
|}

Sun Aug 28 16:10:56 EDT BGQ racks were shutoff due to environment issue, system being restarted. Please resubmit your jobs.

Sun 28 Aug 2016 10:19:17 EDT Turning systems on. Expect to be up in a couple of hours.

Sun Aug 28 09:16:02 EDT 2016 Datacentre was shutdown due to facility power failure.

Tue 16 Aug 2016 21:55:20 Cooling restored, filesystem up and OK, bringing up clusters. GPC should be available to users by 11PM (perhaps as early as 1030)

Tue 16 Aug 2016 20:46:37 Water service has been restored to building. Restarting cooling system.

Tue 16 Aug 2016 19:52:32 SciNet-related maintenance and modifications have been completed successfully. Work on the building water valve is expected to be done on time (by 9PM). Once water service is restored we need to restore cooling, power-up filesystems and then restart the clusters. Unlikely that any systems are available to users before 11PM and it will take longer to get everything online. Check here for updates

Tue 16 Aug 2016 07:07:32 Shutdown has started

 Scheduled full-day maintenance shutdown begins:

7AM, Tuesday, 16 Aug 

Several projects (adding new 208V circuits for storage, cooling tower maintenance etc) are being carried out on same day as the landlord needs to shutdown the main building water supply (and therefore our cooling system as well) for repairs.

Expect to start bringing systems up about 10PM depending on when the water work is done. Check here for further updates during the day

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-28T22:22:00Z

Chen: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up]]File System
|-
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:down.png|down|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|[[File:down.png|down|link=BGQ]][[BGQ]]
|[[File:down.png|down|link=HPSS]][[HPSS]]
|}

Sun Aug 28 16:10:56 EDT BGQ racks were shutoff due to environment issue, system being restarted. Please resubmit your jobs.

Sun 28 Aug 2016 10:19:17 EDT Turning systems on. Expect to be up in a couple of hours.

Sun Aug 28 09:16:02 EDT 2016 Datacentre was shutdown due to facility power failure.

Tue 16 Aug 2016 21:55:20 Cooling restored, filesystem up and OK, bringing up clusters. GPC should be available to users by 11PM (perhaps as early as 1030)

Tue 16 Aug 2016 20:46:37 Water service has been restored to building. Restarting cooling system.

Tue 16 Aug 2016 19:52:32 SciNet-related maintenance and modifications have been completed successfully. Work on the building water valve is expected to be done on time (by 9PM). Once water service is restored we need to restore cooling, power-up filesystems and then restart the clusters. Unlikely that any systems are available to users before 11PM and it will take longer to get everything online. Check here for updates

Tue 16 Aug 2016 07:07:32 Shutdown has started

 Scheduled full-day maintenance shutdown begins:

7AM, Tuesday, 16 Aug 

Several projects (adding new 208V circuits for storage, cooling tower maintenance etc) are being carried out on same day as the landlord needs to shutdown the main building water supply (and therefore our cooling system as well) for repairs.

Expect to start bringing systems up about 10PM depending on when the water work is done. Check here for further updates during the day

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-28T22:21:28Z

Chen: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up]]File System
|-
|[[File:up.png|up|link=Gravity]][[Gravity]]
|[[File:up.png|up|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:down.png|down|link=Visualization Nodes]][[Visualization Nodes|Viz]]
|[[File:down.png|up|link=BGQ]][[BGQ]]
|[[File:down.png|down|link=HPSS]][[HPSS]]
|}

Sun Aug 28 16:10:56 EDT 2016 EDT BGQ racks were shutoff due to environment issue, system being restarted. Please resubmit your jobs.

Sun 28 Aug 2016 10:19:17 EDT Turning systems on. Expect to be up in a couple of hours.

Sun Aug 28 09:16:02 EDT 2016 Datacentre was shutdown due to facility power failure.

Tue 16 Aug 2016 21:55:20 Cooling restored, filesystem up and OK, bringing up clusters. GPC should be available to users by 11PM (perhaps as early as 1030)

Tue 16 Aug 2016 20:46:37 Water service has been restored to building. Restarting cooling system.

Tue 16 Aug 2016 19:52:32 SciNet-related maintenance and modifications have been completed successfully. Work on the building water valve is expected to be done on time (by 9PM). Once water service is restored we need to restore cooling, power-up filesystems and then restart the clusters. Unlikely that any systems are available to users before 11PM and it will take longer to get everything online. Check here for updates

Tue 16 Aug 2016 07:07:32 Shutdown has started

 Scheduled full-day maintenance shutdown begins:

7AM, Tuesday, 16 Aug 

Several projects (adding new 208V circuits for storage, cooling tower maintenance etc) are being carried out on same day as the landlord needs to shutdown the main building water supply (and therefore our cooling system as well) for repairs.

Expect to start bringing systems up about 10PM depending on when the water work is done. Check here for further updates during the day

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-28T19:27:08Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-28T18:20:42Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-28T16:36:52Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-28T13:32:39Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-28T13:32:05Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-28T13:31:45Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-17T04:02:26Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2016-08-17T03:13:08Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2016-04-22T03:23:43Z

Chen:

Oldwiki.scinet.utoronto.ca:System Alerts

2016-04-22T03:23:18Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2016-01-15T16:50:33Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2015-09-24T13:19:25Z

Chen: /* System Status */

File:Mellanox.GD4700.Switch.pdf

2015-03-30T17:43:16Z

Chen:

Oldwiki.scinet.utoronto.ca:System Alerts

2015-02-07T21:19:09Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2015-01-15T16:49:23Z

Chen:

== System Status==

{|
|[[File:up.png| up |link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png| up |link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png| up |link=Sandy]][[Sandy]]
|[[File:up.png| up |link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png| up | up ]]File System
|-
|[[File:up.png| up |link=Gravity]][[Gravity]]
|[[File:up.png| up |link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:down.png| down |link=BGQ]][[BGQ]]
|[[File:down.png| down |link=HPSS]][[HPSS]]
|
|}
Thu Jan 15 11:22:00 EST: Cooling tower fan belt service is finished. Chiller is being serviced as scheduled while the chilled water plant is working on free-cooling mode. We are not expecting any interruption for users. Systems are being brought up now.

Wed Jan 14 17:02:18 EST: '''Emergency shutdown of all compute nodes 8:30AM tomorrow''' (Thurs, 15 Jan). After starting to bring up systems this afternoon we learned that an emergency replacement of the cooling tower fan belt is required tomorrow morning. Compute systems that are currently up will need to be shutdown at 0830 tomorrow. We will attempt to keep login nodes and storage up during tomorrow's downtime which is expected to last 1-4 hrs.

Wed Jan 14 14:34:18 EST: Expect some systems (login nodes, GPC and BGQ) to be available by approx 3:00-3:30PM.

Wed Jan 14 13:09:03 EST: Free-cooling is being restored and should allow compute systems to come online this afternoon. Chiller maintenance will continue throughout the day and possibly into tomorrow. Check back for updates.

''SCHEDULED MAINTENANCE DOWNTIME ANNOUNCEMENT''

On January 14 and 15, '''scheduled maintenance''' on the data centre's cooling system will require all systems to be shut down for at least the first part of the maintenance. All SciNet systems will be shut down at 7 AM on Wednesday January 14, 2015 and all login sessions and jobs will be killed at that time.

At the earliest, the systems will be available again later on Wednesday afternoon, but is it possible that the downtime will extend into Thursday January 15, 2015. Check here on the SciNet wiki (wiki.scinethpc.ca) for updates on Wednesday and Thursday.

--

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2015-01-15T16:26:05Z

Chen: /* System Status */

== System Status==

{|
|[[File:down.png| down |link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png| down |link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:down.png| down |link=Sandy]][[Sandy]]
|[[File:down.png| down |link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png| up | up ]]File System
|-
|[[File:down.png| down |link=Gravity]][[Gravity]]
|[[File:up.png| up |link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:down.png| down |link=BGQ]][[BGQ]]
|[[File:down.png| down |link=HPSS]][[HPSS]]
|
|}
Thu Jan 15 11:22:00 EST: Cooling tower fan belt service is finished. Chiller is being serviced as scheduled while the chilled water plant is working on free-cooling mode. We are not expecting any interruption for users. Systems are being brought up now.

Wed Jan 14 17:02:18 EST: '''Emergency shutdown of all compute nodes 8:30AM tomorrow''' (Thurs, 15 Jan). After starting to bring up systems this afternoon we learned that an emergency replacement of the cooling tower fan belt is required tomorrow morning. Compute systems that are currently up will need to be shutdown at 0830 tomorrow. We will attempt to keep login nodes and storage up during tomorrow's downtime which is expected to last 1-4 hrs.

Wed Jan 14 14:34:18 EST: Expect some systems (login nodes, GPC and BGQ) to be available by approx 3:00-3:30PM.

Wed Jan 14 13:09:03 EST: Free-cooling is being restored and should allow compute systems to come online this afternoon. Chiller maintenance will continue throughout the day and possibly into tomorrow. Check back for updates.

''SCHEDULED MAINTENANCE DOWNTIME ANNOUNCEMENT''

On January 14 and 15, '''scheduled maintenance''' on the data centre's cooling system will require all systems to be shut down for at least the first part of the maintenance. All SciNet systems will be shut down at 7 AM on Wednesday January 14, 2015 and all login sessions and jobs will be killed at that time.

At the earliest, the systems will be available again later on Wednesday afternoon, but is it possible that the downtime will extend into Thursday January 15, 2015. Check here on the SciNet wiki (wiki.scinethpc.ca) for updates on Wednesday and Thursday.

--

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2015-01-15T16:18:16Z

Chen: /* System Status */

== System Status==

{|
|[[File:down.png| down |link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png| down |link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:down.png| down |link=Sandy]][[Sandy]]
|[[File:down.png| down |link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png| up | up ]]File System
|-
|[[File:down.png| down |link=Gravity]][[Gravity]]
|[[File:up.png| up |link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:down.png| down |link=BGQ]][[BGQ]]
|[[File:down.png| down |link=HPSS]][[HPSS]]
|
|}

Wed Jan 14 17:02:18 EST: '''Emergency shutdown of all compute nodes 8:30AM tomorrow''' (Thurs, 15 Jan). After starting to bring up systems this afternoon we learned that an emergency replacement of the cooling tower fan belt is required tomorrow morning. Compute systems that are currently up will need to be shutdown at 0830 tomorrow. We will attempt to keep login nodes and storage up during tomorrow's downtime which is expected to last 1-4 hrs.

Wed Jan 14 14:34:18 EST: Expect some systems (login nodes, GPC and BGQ) to be available by approx 3:00-3:30PM.

Wed Jan 14 13:09:03 EST: Free-cooling is being restored and should allow compute systems to come online this afternoon. Chiller maintenance will continue throughout the day and possibly into tomorrow. Check back for updates.

''SCHEDULED MAINTENANCE DOWNTIME ANNOUNCEMENT''

On January 14 and 15, '''scheduled maintenance''' on the data centre's cooling system will require all systems to be shut down for at least the first part of the maintenance. All SciNet systems will be shut down at 7 AM on Wednesday January 14, 2015 and all login sessions and jobs will be killed at that time.

At the earliest, the systems will be available again later on Wednesday afternoon, but is it possible that the downtime will extend into Thursday January 15, 2015. Check here on the SciNet wiki (wiki.scinethpc.ca) for updates on Wednesday and Thursday.

--

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2015-01-15T15:02:42Z

Chen: /* System Status */

== System Status==

{|
|[[File:down.png| down |link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:down.png| down |link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:down.png| down |link=Sandy]][[Sandy]]
|[[File:down.png| down |link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png| up | up ]]File System
|-
|[[File:down.png| down |link=Gravity]][[Gravity]]
|[[File:down.png| down |link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:down.png| down |link=BGQ]][[BGQ]]
|[[File:down.png| up |link=HPSS]][[HPSS]]
|
|}

Wed Jan 14 17:02:18 EST: '''Emergency shutdown of all compute nodes 8:30AM tomorrow''' (Thurs, 15 Jan). After starting to bring up systems this afternoon we learned that an emergency replacement of the cooling tower fan belt is required tomorrow morning. Compute systems that are currently up will need to be shutdown at 0830 tomorrow. We will attempt to keep login nodes and storage up during tomorrow's downtime which is expected to last 1-4 hrs.

Wed Jan 14 14:34:18 EST: Expect some systems (login nodes, GPC and BGQ) to be available by approx 3:00-3:30PM.

Wed Jan 14 13:09:03 EST: Free-cooling is being restored and should allow compute systems to come online this afternoon. Chiller maintenance will continue throughout the day and possibly into tomorrow. Check back for updates.

''SCHEDULED MAINTENANCE DOWNTIME ANNOUNCEMENT''

On January 14 and 15, '''scheduled maintenance''' on the data centre's cooling system will require all systems to be shut down for at least the first part of the maintenance. All SciNet systems will be shut down at 7 AM on Wednesday January 14, 2015 and all login sessions and jobs will be killed at that time.

At the earliest, the systems will be available again later on Wednesday afternoon, but is it possible that the downtime will extend into Thursday January 15, 2015. Check here on the SciNet wiki (wiki.scinethpc.ca) for updates on Wednesday and Thursday.

--

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2014-12-05T18:13:35Z

Chen: /* System Status */

== System Status==

{|
|[[File:up.png| up |link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png| up |link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png| up |link=Sandy]][[Sandy]]
|[[File:up.png| up |link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png| up | up ]]File System
|-
|[[File:up.png| up |link=Gravity]][[Gravity]]
|[[File:up.png| up |link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png| up |link=BGQ]][[BGQ]]
|[[File:down.png| down |link=HPSS]][[HPSS]]
|
|}

Fri Dec 5 08:25:48 EST 2014: City has just confirmed that water has been restored. Staff are on-site and restarting cooling systems. Users should have access to compute systems before noon.

Fri Dec 5 06:51:56 EST 2014: City has reported that water should be restored by 8AM. If they're on schedule, it could still take a few hours after that to restart cooling systems, power-up and test storage etc

Thu 4 Dec 2014 16:05:07 EST: Some systems are back up, but only until tonight when the advertised shutdown will still happen, starting at around 9PM. Only short jobs, that fit in the short time before the systems are taken down, will run. Filesystems are up, as well as devel nodes on all platforms. BGQ will remain down until after the water repairs.

Thu Dec 4 14:03:24 EST 2014: Systems abnormally shutdown due to loss of plumbing to secondary loop. Still investigating.

ALL SYSTEMS TO BE SHUTDOWN ON THURSDAY- On Thursday Dec 4, at 9PM EST, all systems will need to be shutdown. The city of Vaughan has advised us that the city water supply will be turned off in order to fully fix the problem that occured on Nov 21. With no water supply we cannot cool the datacentre, hence the shutdown. We expect all systems to be back up on Friday Dec 5, at around 11AM.

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2014-10-06T15:07:25Z

Chen:

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png|up]]File System
|-
|[[File:up.png| up |link=Gravity]][[Gravity]]
|[[File:up.png| up |link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png| up |link=BGQ]][[BGQ]]
|[[File:up.png| up |link=HPSS]][[HPSS]]
|
|}

Sun Oct 5 18:19:14 EDT 2014: Systems started coming online 5:40PM. GPC, TCS and BGQ are running jobs again

Sun Oct 5 15:57:55 EDT 2014: Power was restored at 2:30PM. Cooling system has been restored. Filesystems being tested. System availability for users likely by 6PM

Sun Oct 5 13:47:48 EDT 2014: Power has still not been restored to the area. Our 5PM estimate above for restoring systems was based on Powerstream's forecast that they would be done by noon. Once power is back we'll update our estimate

Sun Oct 5 06:25:25 EDT 2014: All systems down as per note above

Mon Sept 29 00:43:00 EDT 2014: HPSS up.

Sun Sep 28 20:00:00 EDT 2014: ARC up and accepting jobs. File systems should be fixed too.

Sun Sep 28 18:30:00 EDT 2014: Sandy and Gravity are up. ARC is up but the schedule is not yet operational from arc01 (use gpc nodes to submit). Some filesystem issues with gss (/scratch2 for group using this) on some gpc nodes are being investigated too.

Sun Sep 28 13:59:00 EDT 2014: BGQ up.

Sun Sep 28 13:37:00 EDT 2014: GPC, P7 and TCS are back up. Users will be able to login shortly. All jobs were killed in the event, so please resubmit. We're working on getting the BGQ, Sandy and Gravity systems up too.

Sun Sep 28 09:43:28 EDT 2014: Brief power outage knocked-out cooling system at about 0806 this morning. Cooling has been restored. Disk controllers and filesystems are being brought up. Systems will be unavailable until at least noon.

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2014-10-06T15:06:30Z

Chen: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:up.png|up|link=Sandy]][[Sandy]]
|[[File:up.png|up|link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png|up]]File System
|-
|[[File:up.png| up |link=Gravity]][[Gravity]]
|[[File:up.png| up |link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png| up |link=BGQ]][[BGQ]]
|[[File:up.png| up |link=HPSS]][[HPSS]]
|
|}

'''SYSTEMS WILL GO DOWN AT 5AM SUNDAY (5 Oct)''' for relocation of hydro services due to nearby construction. Expect to be back up by 5PM but please check here for updates

Sun Oct 5 18:19:14 EDT 2014: Systems started coming online 5:40PM. GPC, TCS and BGQ are running jobs again

Sun Oct 5 15:57:55 EDT 2014: Power was restored at 2:30PM. Cooling system has been restored. Filesystems being tested. System availability for users likely by 6PM

Sun Oct 5 13:47:48 EDT 2014: Power has still not been restored to the area. Our 5PM estimate above for restoring systems was based on Powerstream's forecast that they would be done by noon. Once power is back we'll update our estimate

Sun Oct 5 06:25:25 EDT 2014: All systems down as per note above

Mon Sept 29 00:43:00 EDT 2014: HPSS up.

Sun Sep 28 20:00:00 EDT 2014: ARC up and accepting jobs. File systems should be fixed too.

Sun Sep 28 18:30:00 EDT 2014: Sandy and Gravity are up. ARC is up but the schedule is not yet operational from arc01 (use gpc nodes to submit). Some filesystem issues with gss (/scratch2 for group using this) on some gpc nodes are being investigated too.

Sun Sep 28 13:59:00 EDT 2014: BGQ up.

Sun Sep 28 13:37:00 EDT 2014: GPC, P7 and TCS are back up. Users will be able to login shortly. All jobs were killed in the event, so please resubmit. We're working on getting the BGQ, Sandy and Gravity systems up too.

Sun Sep 28 09:43:28 EDT 2014: Brief power outage knocked-out cooling system at about 0806 this morning. Cooling has been restored. Disk controllers and filesystems are being brought up. Systems will be unavailable until at least noon.

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2014-09-28T17:45:34Z

Chen: /* System Status */

== System Status==

{|
|[[File:up.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:down.png|down|link=Sandy]][[Sandy]]
|[[File:down.png|down|link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up.png|up]]File System
|-
|[[File:down.png|down|link=Gravity]][[Gravity]]
|[[File:up.png|down|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:up.png|down|link=BGQ]][[BGQ]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|
|}

Sun Sep 28 13:37:00 EDT 2014: GPC, P7 and TCS are back up. Users will be able to login shortly. All jobs were killed in the event, so please resubmit. We're working on getting the BGQ, Sandy and Gravity systems up too.

Sun Sep 28 09:43:28 EDT 2014: Brief power outage knocked-out cooling system at about 0806 this morning. Cooling has been restored. Disk controllers and filesystems are being brought up. Systems will be unavailable until at least noon.

Fri Sep 19 15:50:54 EDT 2014: Scheduler has been stable for the past hour, and jobs are being scheduled. Please submit your jobs. Please be aware that showq is not reporting some running jobs that were running before the glitch. Use qstat instead of showq for these jobs. Most of queued jobs this morning were rejected by the scheduler when it went back online.

Fri Sep 19 12:52:20 EDT 2014: We've been experiencing intermittent problems with the scheduler. Job submission has been paused temporarily, until we can restart the scheduler.
Please check this space for updates.

Mon Sep 15 11:13:10 EDT 2014: The scheduler had some issues this morning and had to be restarted to resolve them. Some queued and running jobs have been lost.

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2014-09-28T17:35:43Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2014-09-28T17:35:13Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2014-09-28T16:43:21Z

Chen: /* System Status */

== System Status==

{|
|[[File:down.png|up|link=GPC Quickstart]][[GPC Quickstart|GPC]]
|[[File:up.png|up|link=TCS Quickstart]][[TCS Quickstart|TCS]]
|[[File:down.png|down|link=Sandy]][[Sandy]]
|[[File:down.png|down|link=GPU Devel Nodes]][[GPU Devel Nodes|ARC]]
|[[File:up75.png|up]]File System
|-
|[[File:down.png|down|link=Gravity]][[Gravity]]
|[[File:up.png|down|link=P7 Linux Cluster]][[P7 Linux Cluster|P7]]
|[[File:down.png|down|link=BGQ]][[BGQ]]
|[[File:down.png|up|link=HPSS]][[HPSS]]
|
|}

Sun Sep 28 09:43:28 EDT 2014: Brief power outage knocked-out cooling system at about 0806 this morning. Cooling has been restored. Disk controllers and filesystems are being brought up. Systems will be unavailable until at least noon.

Fri Sep 19 15:50:54 EDT 2014: Scheduler has been stable for the past hour, and jobs are being scheduled. Please submit your jobs. Please be aware that showq is not reporting some running jobs that were running before the glitch. Use qstat instead of showq for these jobs. Most of queued jobs this morning were rejected by the scheduler when it went back online.

Fri Sep 19 12:52:20 EDT 2014: We've been experiencing intermittent problems with the scheduler. Job submission has been paused temporarily, until we can restart the scheduler.
Please check this space for updates.

Mon Sep 15 11:13:10 EDT 2014: The scheduler had some issues this morning and had to be restarted to resolve them. Some queued and running jobs have been lost.

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2014-09-28T12:20:42Z

Chen:

Oldwiki.scinet.utoronto.ca:System Alerts

2014-09-28T12:20:17Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2014-08-01T23:38:43Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2014-08-01T22:54:24Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2014-06-30T21:52:16Z

Chen:

Oldwiki.scinet.utoronto.ca:System Alerts

2014-06-30T21:26:45Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2014-01-10T09:46:51Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2014-01-10T09:45:34Z

Chen: /* System Status */

Oldwiki.scinet.utoronto.ca:System Alerts

2013-01-17T22:23:27Z

Chen: /* System Status: UP (partially) */

== System Status: '''UP'''==

Thu Jan 17 17:21:01 EST 2013: 
Chiller maintenance work finished. System is running normally.

Last updated Thu Jan 17 17:21:01 EST 2013 
([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2012-07-26T14:47:23Z

Chen:

== System Status: '''NORMAL'''==

Thu Jul 26 10:45:37 EDT 2012: Systems back online. Please report any problems/issue to support@scinet.utoronto.ca

Thu Jul 26 06:49:51 EDT 2012: Power lost at 0619 this morning.

 
([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2012-07-26T14:45:59Z

Chen: /* System Status: DOWN */

== System Status: '''NORMAL'''==

Thu Jul 26 10:45:37 EDT 2012: System UP.

Thu Jul 26 07:58:25 EDT 2012: Staff on-site. Power has been restored. Cooling systems being brought online.

Thu Jul 26 06:49:51 EDT 2012: Power lost at 0619 this morning. Because of timing and vacation schedules it will likely be noon-ish before system access is restored.

 
([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2012-07-08T15:50:19Z

Chen: /* System Status: DOWN */

== System Status: '''UP'''==

Sun Jul 8 11:49:20 EDT 2012 Systems back online after power failure at Jul 8 09:17:13. Report any problems to support@scinet.utoronto.ca
([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2012-07-08T13:26:36Z

Chen: /* System Status: UP */

== System Status: '''DOWN'''==

Jul 8 09:17:13 Power Failure. ([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2012-04-12T21:34:57Z

Chen: /* System Status: TCS UP/GPC AND OTHER SYSTEMS UP */

== System Status: '''ALL SYSTEMS UP'''==
Thu Apr 12 17:08:00 EST 2012: scheduled maintenance downtime of the TCS. As announced, running TCS jobs and TCS login sessions were killed. All other systems are up. The TCS is expected to be up again sometime this evening.

--------------------------------

Tue Apr 10 16:24:00 EST 2012: ''scheduled downtimes:''

Apr 12: TCS Shutdown (Other systems will remain up). The shutdown will start at 11 am and the system should be available again at in the evening of the same day.

Apr 18-19: Full SciNet shutdown. More details later.

------------------------------

Thu Feb 9 11:50:57 EST 2012: ''System Temporary Change for MPI ethernet jobs:''

Due to some changes we are making to the GPC GigE nodes, if you run multinode ethernet MPI jobs (IB multinode jobs are fine), you will need to explicitly request the ethernet interface in your mpirun:

For Openmpi -> mpirun --mca btl self,sm,tcp 
For IntelMPI -> mpirun -env I_MPI_FABRICS shm:tcp

There is no need to do this if you run on IB, or if you run single node mpi jobs on the ethernet (GigE) nodes. Please check [[GPC_MPI_Versions]] for more details.

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2012-04-12T21:28:49Z

Chen: /* System Status: TCS DOWN/GPC AND OTHER SYSTEMS UP */

== System Status: '''TCS UP'''/'''GPC AND OTHER SYSTEMS UP'''==
Thu Apr 12 17:08:00 EST 2012: scheduled maintenance downtime of the TCS. As announced, running TCS jobs and TCS login sessions were killed. All other systems are up. The TCS is expected to be up again sometime this evening.

--------------------------------

Tue Apr 10 16:24:00 EST 2012: ''scheduled downtimes:''

Apr 12: TCS Shutdown (Other systems will remain up). The shutdown will start at 11 am and the system should be available again at in the evening of the same day.

Apr 18-19: Full SciNet shutdown. More details later.

------------------------------

Thu Feb 9 11:50:57 EST 2012: ''System Temporary Change for MPI ethernet jobs:''

Due to some changes we are making to the GPC GigE nodes, if you run multinode ethernet MPI jobs (IB multinode jobs are fine), you will need to explicitly request the ethernet interface in your mpirun:

For Openmpi -> mpirun --mca btl self,sm,tcp 
For IntelMPI -> mpirun -env I_MPI_FABRICS shm:tcp

There is no need to do this if you run on IB, or if you run single node mpi jobs on the ethernet (GigE) nodes. Please check [[GPC_MPI_Versions]] for more details.

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2011-12-28T07:33:53Z

Chen: /* System Status: UP */

== System Status: '''DOWN''' ==

Datacentre lost power.

Wed Dec 28 02:33 EST 2011

''Note: From Dec 21, 2011 to Jan 1, 2012, the SciNet offices are officially closed, but the system will be up and running and we will keep an eye out for emergencies.''

([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2011-06-09T05:03:09Z

Chen: /* System Status: DOWN */

== System Status: NORMAL ==

Last updated: Thu Jun 9 00:32:55 EDT 2011 ([[Previous_messages:|Previous messages]])

Oldwiki.scinet.utoronto.ca:System Alerts

2011-05-17T21:30:23Z

Chen: /* System Status: Chiller Problem, datacentre PARTIALLY UP */

== System Status: NORMAL ==

Cause of the last two chiller outages appears to have been found. Hope to get repairs done tomorrow (Tuesday). In the meantime 1/3 of the GPC and a couple of TSC nodes are running with the datacentre in free-cooling mode.

You can check our twitter feed, @SciNetHPC, for updates.

Last updated: Mon May 17 17:30:00 EDT 2011 ([[Previous_messages:|Previous messages]])