DVR system locking up can't web in to manage

Hello,

I have my Blue Cherry as a VM on ESXI 7.0. I am running Ubuntu 20.04.5 LTS. I do not know if it is a Ubuntu update or what that causes it to lock up. I am not able to log into the web management interface.
I reboot and still the same issue, can not web into the web management interface.
I do get the logon page and put in user name and password and then it just sits and thinks as it tries to log in. Then web page after a number of minutes comes up with Could not connect: [111] Connection refused. I did even check with the Linux client. When I find the system locked up the client also can not connect to the DVR server. I hope the uploaded photo works and is able to be seen.

I just tried the web management again and got the 504 gateway times. out. I checked the Linux client and it connects and shows active and appears to be recording. I am using continuous record on all cameras. But the Linux client is up and communicating, I can see the camera views. I will be setting a Cron job to reboot every 24 hours to help make sure the DVR is at least alive and recording. Not sure why the web management is not working. Just checked and UFW firewall is inactive so not being blocked that way.

Origional error message shown below.

I just ran apt-get update / upgrade and seen it was installing Bluecherry 3.3.1.0RC7, noticed this message as it was installing.

Rebooted server Everything seems to be working and I can now log into the web management. But 2 weeks ago I was able to do that and then yesterday and today I was not able to do so after reboots. I will see if rc7 in a week or two have the same issues where I can not log into the web management.
I will check back and update this in a week or two. Next to get the reboot cron set. Running in continuous mode with like 18 cameras on a VM with 4 cores or CPUs I am running 35% CPU usage with 2265MB out of 16 Gig ram allocated. When the DVR is working, It works very, very well! Looking forward to when the stable DVR for Ubuntu 22.04 LTR is released.

Chad

Looks to be the same issue I’ve been having - I had the issue here as described - Error: Could not connect: [111 ] Connection refused. I applied the fix advised but didn’t seem to work - Kept locking up usually about every 1-2 days - I can only assume it was when the drive was getting full and perhaps removing old recordings etc.

It was stable for about 2 weeks then the bug came back. Ashame, it’s not possible to rollback MariaDB to see if that’s the issue… What version of MariaDB are you running?? My version is below.

As part of my troubleshooting I even re-built my NVR.

I’m running,

PRETTY_NAME=“Debian GNU/Linux 11 (bullseye)”
NAME=“Debian GNU/Linux”
VERSION_ID=“11”
VERSION=“11 (bullseye)”
VERSION_CODENAME=bullseye
ID=debian
HOME_URL=“https://www.debian.org/
SUPPORT_URL=“Debian -- User Support
BUG_REPORT_URL=“https://bugs.debian.org/

16GB RAM / 11 Cameras / On Camera Motion Detection

MariaDB --version - mariadb Ver 15.1 Distrib 10.5.18-MariaDB, for debian-linux-gnu (x86_64)

RC7 has been released for Ubuntu 20.04 and 22.04 which should resolve this. Bullseye RC7 should be out this week for testing.

Hello,

Well its been a week since I posted my information. I am running DVR Version 3.1.0-rc7. I also did add a cron job to reboot the entire server every night. So I assume that is also maybe helping. Though not really sure, because before applying updates & rc7 and everything to the Ubuntu server (Linux and Bluecherry updates.) a reboot would not fix the issue. As Curtis noted, I assume the rc7 it working well and my nightly reboot is moot point, but I am leaving it in. My System is running on ESXI 7.0.3 I think patch J or something. So will keep an eye on things. Before this post I thought I was running the rc7. Thank you Curtis for the update info.

Cool Ubuntu 22.04 be out soon. I will plan to update to that, build a new VM fully configured for that and then can make my OVF file for backup with everything configured and ready to go. Had to entirely rebuild and add additional capacity to the ESXI server last month and lucky I had OVF files of my critical VM machines and made doing a restore of the entire VM an easy quick process.

Hello,

I am having the same error again with the DVR web interface / web log in. Since I last replied in this, the same issue has again happen. I have uploaded the error i get in the browser. My Ubuntu DVR client app logs into the server fine and is functioning. and everything from the DVR perspective is the system is working and recording. I just can’t web into the web/management side of it.

I have GUI on Ubuntu installed as well as using webmin to access and manage the Ubuntu system. When I had it pull updates I did not see any updates or anything with Bluecherry. Not sure if any Ubuntu updates will fix anything of it I will have to try the commands to try to fix the issue. The system is running the latest version of the DVR as I had documented in my previous posts in this thread. I am on RC7. Ubuntu updates being still applied. Will see after completion and reboot.

I am running updates again just now on it.

Can we get remote access to this server to see where the problem lies? If so please PM me.

Hello,

I will see about getting things on the fire wall set. I was having issues after updating. I did a reboot and purged out a bunch of recordings. I dropped the disk usage to 1% and let the system purge out or delete recordings or how ever it works when it reaches it’s max % disk space used. If seems to be working okay for the moment. Confirmed I am running 3.1.0-rc7. At first it looked like there was a memory leak. (F.Y.I. this is running on ESXI 7. something VMware. As soon as I get firewall set for SSH I will let you know. Last notes from 1-11-23 when I last rebuilt it, everything was at defaults. Confirmed Admin account is still active and at defaults of bl…

Are you using network storage and / or continuous recording?

The next time this happen can you attach the output of these:

The next time this happens can you run:

​cat /proc/meminfo
free
vmstat

and a screenshot or output of ‘htop’ (preferred method below):

sudo apt install htop aha html2text
echo q | htop -C | aha --line-fix | html2text -width 999 | grep -v “F1Help” | grep -v “xml version=” > file.txt

Hello, I am using local internal storage, and I am using continuous recording. Tuesday 3-7-23 I will be available late morning and all afternoon, should you want to try and have an interactive session / call.
I believe the issue is still ongoing, I will run the commands and get back to you.
Chad

ssh is preferred access for us. Some of our developers work in different time zones and it’s counterproductive to schedule a zoom or team viewer as we need repeated console (ssh) access to check logs and other items over the course of a few days.

If you prefer we can do reverse ssh access so you don’t have to open a firewall.

Hello,

I will enable ssh Sunday night for the DVR system.

Yes I am still getting the error. I will get the logon page, after I authenticate the very next page displays the "Could not connect: [111] Connection refused "

​had@VMDVR:~$ cat /proc/meminfo
MemTotal: 65857272 kB
MemFree: 63521528 kB
MemAvailable: 64261288 kB
Buffers: 110584 kB
Cached: 1192828 kB
SwapCached: 0 kB
Active: 556112 kB
Inactive: 1477648 kB
Active(anon): 1640 kB
Inactive(anon): 734028 kB
Active(file): 554472 kB
Inactive(file): 743620 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 2097148 kB
SwapFree: 2097148 kB
Dirty: 128 kB
Writeback: 0 kB
AnonPages: 730348 kB
Mapped: 309524 kB
Shmem: 10704 kB
KReclaimable: 69300 kB
Slab: 128352 kB
SReclaimable: 69300 kB
SUnreclaim: 59052 kB
KernelStack: 8192 kB
PageTables: 12752 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 35025784 kB
Committed_AS: 3618724 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 25820 kB
VmallocChunk: 0 kB
Percpu: 1608 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 214912 kB
DirectMap2M: 66893824 kB

chad@VMDVR:~$ free
total used free shared buff/cache available
Mem: 65857272 973296 63511196 10704 1372780 64251016
Swap: 2097148 0 2097148
chad@VMDVR:~$

chad@VMDVR:~$ vmstat
procs -----------memory---------- —swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 63511196 110616 1262164 0 0 12 9 113 205 0 0 99 0 0
chad@VMDVR:~$

sudo apt install htop aha html2text - all installed fine.

Below does not run.
echo q | htop -C | aha --line-fix | html2text -width 999 | grep -v “F1Help” | grep -v “xml version=” > file.txt

root@VMDVR:/home/chad# echo q | htop -C | aha --line-fix | html2text -width 999 | grep -v “F1Help” | grep -v “xml version=” > file.txt
grep: version=”: No such file or directory
root@VMDVR:/home/chad#

Get this message:
oot@VMDVR:/home/chad#
root@VMDVR:/home/chad# sudo echo q | htop -C | aha --line-fix | html2text -width 999 | grep -v “F1Help” | grep -v “xml version=” > file.txt
grep: version=”: No such file or directory
root@VMDVR:/home/chad#

Just ran apt-get update and then upgrade again and installed all updates.

Now it is running and working after running an Ubuntu patch up[date. Just now 3-4-23 I did a full update and reboot and I can now log into the user interface / management on port 7001. I do have a cron job set to reboot every 24 hours still in place.
DVR-Userinterface3-4-23

HUMMMMMMMMMMM Its them chupacabras I tell you!!! Well will see how long it holds. If you still want to poke around, let me know, I will still open up SSH Sunday night so Monday it should be all set. DO I need to create a specific user account?

By the way, do you have any specific recommendations for memory and CPU’s / CPU cores I should configure for 20 cameras? Currently I have 2 CPU’s, 64 gigs ram assigned to this VM. At that, DVR web interface shows I am running about 39% of the Virtual machine’s CPU, so based on that, looks like I have a good balance. I am out of Host CPUs / cores for the moment. Looking to see where the 12 core CPUs are priced and maybe add some more cores so I can put 4 or 6 cores to the DVR system. Will be adding a few more cameras soon.

Chad

This should be sufficient. The next time the web UI freezes please let me know…and also see if port 7003 is open. This port is internal only and should not be opened to the outside world.

telnet 0 7003

AND

sudo apt install net-tools
sudo netstat -an | grep 7003

Thank you. I will check that out. I am sure it will happen again. It has on and off. Very odd. I just have it set and it runs, I really don’t access it or anything but once a week to just run updates. Come to think of it I should take a snapshot of it as it is now that I might be able to restore back to. Thank you for all the assistance.

Chad

Hello, I was shutting down cameras and getting ready to shut down for taking a snapshot and I see it lost my license / activation. What might be causing that? What does it show on your side for my license / activation status?

Well it popped the same error again, I shut off cameras / disabled them in the management port 7001 interface. Rebooted server and now have same error.

same-error-3-4-23

root@VMDVR:/home/chad# apt install net-tools
Reading package lists… Done
Building dependency tree
Reading state information… Done
net-tools is already the newest version (1.60+git20180626.aebd88e-1ubuntu1).
The following packages were automatically installed and are no longer required:
gir1.2-goa-1.0 libfwupdplugin1 libxmlb1 linux-headers-5.13.0-39-generic linux-hwe-5.13-headers-5.13.0-39 linux-image-5.13.0-39-generic linux-modules-5.13.0-39-generic
linux-modules-extra-5.13.0-39-generic
Use ‘apt autoremove’ to remove them.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
root@VMDVR:/home/chad# sudo netstat -an | grep 7003
root@VMDVR:/home/chad# netstat -an | grep 7003
root@VMDVR:/home/chad# ufw status
Status: inactive
root@VMDVR:/home/chad#

oot@VMDVR:/home/chad# telnet 0 7003
Trying 0.0.0.0…
telnet: Unable to connect to remote host: Connection refused
root@VMDVR:/home/chad#

I tried the telnet as root and as just me, same error.
I will have SSH open on firewall tomorrow night. Ubuntu firewall is disabled. and my UTM will point SSH to the DVR system.

Sorry, I told you the wrong port. Please try 7004

sudo netstat -an | grep 7004

Hello, My system fully crashed, I thought I would restore from a backup and inadvertently restored a 3RC5 and did not have a 3RC7 backup. So that is why if anyone tried to ssh in to my system it was not working, Sunday night / Monday for some odd reason I tried to get back to this forum and got the 500 Bad Gateway message. I need to just reload Ubuntu from scratch and then install the DVR but appears the links when I tried are not working. What is the correct path and url / script to run to get DRV3RC7 installed? When I would run the apt-get update it was having issues finding the repo. I am trying to see if online chat from the website will get me in touch with some one.

dvr-apt-update-2-28-23
.

Chad P.

What a fun few past days! System is back up and online. Ubuntu rebuilt, DVR running 3.1.0-rc7.
I just need the License reset, if you could when you have time Curtis. Well for now everything is running, will see if the issue pops back up or not. Ubuntu is fully patched and all and I did make / run a system config backup file so I have that to restore from if needed. Thank you for all the assistance.

Sincerely, Chad