I’m running Bluecherry 3.1.14 on Ubuntu Linux 6.17.0-22-generic #22~24.04.1-Ubuntu
I have not had any issues for a while now, then today I noticed my recordings stopped, I did a reboot of the computer and they started back up but adding in a new camera and setting motion detection, nothing was happening. I could live vie the camera but no motion detection.
I reviewed the log and found two database errors:
CRITICAL: Database lock timeout - potential deadlock detected
CRITICAL: Database lock retry failed - server may be overloaded
I used the Brave browser AI to see if it could give me some insights and it had me run a few commands, one of them SHOW ENGINE INNODB STATUS\G
I provided the results back and it told me that my InnoDB redo log is completely full (Log capacity used 104857600 out of 104857600 ), which prevents any further write operations and will cause the database to hang or crash.
It gave an instruction to resolve it, which I elected to do:
edit /etc/mysql/mysql.conf.d/mysqld.cnf and change or add
[mysqld]
innodb_redo_log_capacity = 1073741824
I added the setting, there was not an existing setting. and restarted mysqld and bluecherry.
Things are running better but not perfect.
The new camera is functioning with motion detection and recordings, however some recordings are not accessible with an error “Error loading this resource”. Also fdisk management is not working well. I have it set to 90% and when it cleans it attempts to go to 89% but its only removing a handfull of files and having to do file reduction on a regular basis. It appears the database is no longer in sync with the stored recordings and the sync database is not doing anything.
Is there anyway to get the database events and recordings between the database and the stored files back into harmony?
I utilized Claude AI to help some. Claude was able to identify over a million ghost records in the database and every record had the wrong file size for the files that were managed. And the database indicated over 61TB of disk spaced was used on my 5TB disk
Got the records cleaned up, updated all the file sizes corrected. Now the bluecherry data base matches number of files on the disk and the disk storage being used.
Space management is still firing which is set for 90% disk usage but is sitting at 91% usage. When file clean fires off, it only cleans small number of files and never gets down to the expected 89%.
Will watch to see what happens over time.
I’m still getting recorded events with a file that has the error “Error loading this resource”. Not all files for a given camera are doing this, just some of them. After looking at this, it looks like bluecherry is cleaning up files from today and not the oldest files and that is why I’m possibly getting Error resource load error.
On a camera that was added in during all the issues, this one I’m getting some good and some loading errors for motion events and I can’t get Continuous recording capturing. I have re-
saved all settings since the database cleaen-up but still no continuous recordings.
Looks like the database has lost information on the oldest files to do the purging from and now I’m not getting any recordings.
After noticing only todays files were getting cleared out rather than the oldest, I went back to Claude AI.
Ended up clearing the Archive flag on all the file records. Once this was done Bluecherry almost immediatly started a disk clean cycle and started clearing a lot of files and correctly dropped the disk space down to the desired amount.
Went back a checked and Bluecherry started resetting the archive bit, looks to be part of the normal process but possibly got corrrupted when I had the INNODB issue. Hopefully clearing the archive caused bluecherry to re-evaluate all the records and properly start data management.
Time will tell, right now its sit and wait - hopefully Continuous Recordings will start back up as they seem to have stopped around the time the INNODB issue raised as well.
After the archive reset, the database appears health. Bluecherry is making recordings, even the continuous recordings have resumed, so problem appears to now have been corrected.
I get a lot of motion events because my cameras are looking at frontyard, backyard, etc and get a lot of wind blowing foliage creating “false” triggers.
Others may wish to review their INNODB setting and increase it to prevent a similar issue. I’m not an expert in any sense, however two different AI’s did mention that the baseline setting of INNODB is too small for this type of application. Since AI’s can make errors, take this with a grain of salt.