MAC36NL - Station - Station crash: powering off and on
Issue
A MAC36NL-family controller’s station has crashed and bringing it back to the running state requires powering the controller off and on.
Possible causes and solutions
First thing to do in this situation is to go through the logs from the Application Director console in platform. Possible causes can have different backgrounds and solutions.
Versions of JAR modules and distribution files are not compliant
Accidental overwriting of basic JAR modules with modules from older Niagara
Occasional stopping of the station while browsing the station
Versions of JAR modules and distribution files are not compliant
Quite often, on controllers of the MAC36NL family, JAR modules (primarily basic ones such as baja-rt
) are installed in different version than the Niagara distribution files (daemon). Such instances are generally unpredictable but there are known cases of accidental station crashing.
All supported Niagara versions are listed in the latest Release Notes:
https://support.ismacontrolli.com/02%20Small%20%26%20Medium%20BMS/iSMA-B-MAC36/Release%20Notes/
Using the wrong version of Workbench
This case occurs if the Workbench Niagara update is different than supported by the MAC36NL controller. For example, the latest Niagara version supported on the MAC36NL is 4.11.1.16 and the Workbench is updated to the newest Niagara 4.11 version, which is 4.11.2.18. Distribution files are stored in a common folder for all Workbench 4.11 instances, allowing Workbench to find distribution files of the MAC36NL controller and perform a commissioning process. However, if the versions are not compliant, all JAR modules are of 4.11.2.18 version, while the controller’s distributions files are of 4.11.1.16 version. This discrepancy may result in a station crash.
Solution:
clean distribution (
cleanDist
) usingplatform
->Distribution File Installer
;install Workbench version supported by the MAC36NL controller, list of supported versions available at: https://support.ismacontrolli.com/Software%20%26%20Tools/Niagara/Installs/;
reopen BAT script from
iSMA_MAC36_files_installer_vX.Y.zip
,reopen newly installed Workbench (important step - Workbench loads files only during opening);
perform controller’s commissioning.
Accidental overwriting of basic JAR modules with modules from older Niagara
While updating the Niagara version, for example, from 4.9 to 4.11, it is not uncommon to move along all useful JAR modules, which are not provided by Tridium by default. To this end, usually all contents of the modules folder from Workbench 4.9 are copied to the modules folder in Workbench 4.11. In this moment, Windows Explorer detects the duplication of modules between the source and destination folders and asks whether to overwrite them. If - by mistake - overwriting is confirmed, instead of choosing not to copy duplicates, 4.11 modules get overwritten with 4.9 modules (including baja-rt
). Therefore, each commissioning means installing 4.11 distribution and 4.9 modules.
Solution:
clean distribution (
cleanDist
) usingplatform
->Distribution File Installer
;uninstall Workbench and remove all remaining files;
reinstall Workbench;
reopen BAT script from
iSMA_MAC36_files_installer_vX.Y.zip
;reopen newly installed Workbench (important step - Workbench loads files only during opening);
perform controller’s commissioning.
Problem with schedules
The Application Director
in platform
shows the following logs:
SEVERE [11:40:14 31-Dec-23 AWST][niagaraDriver] Error receiving msg: /Drivers/NiagaraNetwork/MAC36NL/schedules
java.lang.IllegalArgumentException: Subordinate schedule not found for supervisor id: slot:/Schedules/Schedule1
at javax.baja.driver.schedule.BScheduleDeviceExt.processExport(BScheduleDeviceExt.java:224)
at com.tridium.nd.schedule.BScheduleChannel.process(BScheduleChannel.java:147)
at com.tridium.fox.sys.BFoxConnection.process(BFoxConnection.java:452)
at com.tridium.fox.session.SessionDispatcher.dispatch(SessionDispatcher.java:84)
at com.tridium.fox.session.SessionDispatcher.run(SessionDispatcher.java:63)
at java.lang.Thread.run(Thread.java:748)
java.lang.IllegalArgumentException: Subordinate schedule not found for supervisor id: slot:/Schedules/Schedule1
at javax.baja.driver.schedule.BScheduleDeviceExt.processExport(BScheduleDeviceExt.java:224)
at com.tridium.nd.schedule.BScheduleChannel.process(BScheduleChannel.java:147)
at com.tridium.fox.sys.BFoxConnection.process(BFoxConnection.java:452)
at com.tridium.fox.session.SessionDispatcher.dispatch(SessionDispatcher.java:84)
at com.tridium.fox.session.SessionDispatcher.run(SessionDispatcher.java:63)
at java.lang.Thread.run(Thread.java:748)
Controller’s schedules were previously loaded by Supervisor using FOX and connected with local schedules in the Supervisor’s station.
Schedules ORD has been changed
It is possible that after integration of the MAC36NL controller’s schedules to the Supervisor’s station, controller’s station structure gets altered in time. For example, schedules are moved to other folders or the station’s name is changed. In such a case, it is required to remove the Supervisor’s configuration of import and export of schedules and recreate it.
Occasional stopping of the station while browsing the station
In this situation, it is highly recommended to look through the logs in the Application Director
console in platform
, not only the latest ones but the historic ones too.
Lost connection with e-mail provider
In the console, the following logs appear periodically:
javax.mail.AuthenticationFailedException: 535 5.7.8 Error: authentication failed: authentication failure
at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:879)
at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:800)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:714)
at javax.mail.Service.connect(Service.java:372)
at javax.mail.Service.connect(Service.java:231)
at javax.mail.Service.connect(Service.java:180)
at javax.baja.email.BOutgoingAccount.pollQueue(BOutgoingAccount.java:819)
at javax.baja.email.BOutgoingAccount.poll(BOutgoingAccount.java:765)
at javax.baja.email.BEmailAccount.doPoll(BEmailAccount.java:620)
at javax.baja.email.BEmailAccount.access$200(BEmailAccount.java:40)
at javax.baja.email.BEmailAccount$Poller.run(BEmailAccount.java:673)
SEVERE [22:04:10 02-Aug-23 CEST][email] BEmailAccount could not poll:
javax.mail.AuthenticationFailedException: 535 5.7.8 Error: authentication failed: authentication failure
at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:879)
at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:800)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:714)
at javax.mail.Service.connect(Service.java:372)
at javax.mail.Service.connect(Service.java:231)
at javax.mail.Service.connect(Service.java:180)
at javax.baja.email.BOutgoingAccount.pollQueue(BOutgoingAccount.java:819)
at javax.baja.email.BOutgoingAccount.poll(BOutgoingAccount.java:765)
at javax.baja.email.BEmailAccount.doPoll(BEmailAccount.java:620)
at javax.baja.email.BEmailAccount.access$200(BEmailAccount.java:40)
at javax.baja.email.BEmailAccount$Poller.run(BEmailAccount.java:673)
The last log indicates the occurrence of the ENGINE WATCHDOG
mechanism and, additionally, dumping of all threads (thread dump
):
ENGINE WATCHDOG TIMEOUT STACK DUMP @ Wed Aug 02 22:37:20 CEST 20232023-08-02 22:37:20
Full thread dump OpenJDK Client VM (25.181-b122 mixed mode):
There are a few possible reasons for a station to crash in such a situation:
lost Internet connection in the MAC36NL controller - in this case, check the following:
configuration of the network’s infrastructure - outside connections may have been limited by the IT administration,
configuration of the controller’s network in the
TCP/IP Configuration
inplatform
(primarily, IPv4 gateway or DNSv4 servers if manually configured);
incorrect configuration of the e-mail provider in the
IncomingAccount
orOutgoingAccount
components:verify the correct configuration of slots:
Hostname
,Port
,Account Password
,Store
(orTransport
),Use Ssl
,Use Start Tls
, andUse Athentication
, against the data of the e-mail provider, for example, there could have been a change of password forced one a year;
no configuration of the
IncomingAccount
orOutgoingAccount
components.
EmailService provided by Tridium has a known error, which causes a station to crash (over time). The reason here is the EmailService added to the station failing to establish a connection with the e-mail provider when the component is not configured, incorrectly configured, or the controller cannot connect to the given provider. It is recommended to remove all configuration of the EmailService rather than waiting for a client to provide configuration data of the e-mail account.
Too many components of Program type
A common way to reuse the same logic in multiple applications is to write it using a JAVA code rather than using logic blocks. Often in such situations, the Program component from the program palette is used. The possible problem here can be driven by the fact that the controllers are equipped with much weaker microcontrollers than PCs or server units. At the same time, each program component opens a new thread and interprets the code written in it on the fly. Unfortunately, this is not an optimal approach and with a number of Program components used it does not guarantee a stable performance of the station. Given the relatively low computing power, it is not recommended to use over 10 Program components in the station in MAC36NL controllers. It is recommended to follow one of the two optimal solutions:
write own module with a component containing recurring logic and compile it to a JAR file - then, use it in the station instead of Program components (more efficient solution);
prepare a logic template using the TemplateService and reuse it in the station (in case of a mistake in the logic, change is required only in the template, which then updates all its references).
Attempting to save the station ends its operation
In the Application Director
console, the following logs appear after each attempt to save the station:
INFO [08:25:28 31-May-23 CEST][sys] Saving station...
SEVERE [08:26:09 31-May-23 CEST][platform] Error connecting to local session
javax.baja.security.AuthenticationException
at com.tridium.platform.daemon.BDaemonSession.connectionFailed(BDaemonSession.java:3257)
at com.tridium.platform.daemon.BDaemonSession.handleUnauthorizedResponse(BDaemonSession.java:3217)
at com.tridium.platform.daemon.BDaemonSession.handleErrorResponses(BDaemonSession.java:2772)
at com.tridium.platform.daemon.BDaemonSession.getConnection(BDaemonSession.java:2559)
at com.tridium.platform.daemon.BDaemonSession.initHostProperties(BDaemonSession.java:952)
at com.tridium.platform.daemon.BDaemonSession.connect(BDaemonSession.java:290)
at com.tridium.platform.daemon.BDaemonSession.connect(BDaemonSession.java:278)
at com.tridium.platform.daemon.LocalSessionUtil.getLocalSession(LocalSessionUtil.java:166)
at com.tridium.platform.daemon.LocalSessionUtil.localDaemonIsAvailable(LocalSessionUtil.java:389)
at com.tridium.platform.BSystemPlatformService$SystemPlatformServiceSaveListener.stationSaveOk(BSystemPlatformService.java:3117)
at com.tridium.sys.station.Station.saveOk(Station.java:748)
at com.tridium.sys.station.Station.saveSync(Station.java:697)
at com.tridium.sys.station.Station.saveSync(Station.java:565)
at com.tridium.sys.station.BStationSaveJob.run(BStationSaveJob.java:59)
at javax.baja.job.BSimpleJob$JobThread.run(BSimpleJob.java:85)
Caused by: com.tridium.platform.daemon.DaemonAuthenticationException
at com.tridium.platform.daemon.BDaemonSession.connectionFailed(BDaemonSession.java:3256)
... 14 more
INFO [08:26:09 31-May-23 CEST][sys] Saved /home/niagara/niagara_user_home/stations/S16628_UC140/config.bog (41080ms)
The logs indicate that during the saving attempt there has been an exception during a FOX communication authorization with the controller’s platform.
Such exception may occur if the NiagaraNetwork component is removed from the Config/Drivers location.
Solution
Add the NiagaraNetwork component from the niagaraNetwork module to the Config/Drivers location.