Memory leak after update to 3.1.2 on ubuntu 22lts

11

I have also upgraded for Debian 12 as suggested but it did not seem to work and I have the same memory issue. 1 thing noted as I did the upgrade, rsyslog was needed but was not installed so the upgrade could not complete until I installed rsyslog, then the upgrade went fine.

12

Are you using 3.1.3?

13

Sorry fir the dealy, I am using the file that was in the link you provided. 3.1.2 is what s displayed.

14

It looks like I also have the errors that the Ubuntu user reported. systemctl status bluecherry shows these lines. This is a sample. Other cameras also have the same messages:

Sep 05 17:45:33 CamSVR bc-server[52775]: I(3/Driveway): Last message repeated 78 times!
Sep 05 17:45:33 CamSVR bc-server[52775]: I(3/Driveway): Making a snapshot
Sep 05 17:45:34 CamSVR bc-server[52775]: E(3/Driveway): avcodec_send_frame: snapshot encoding failed: Invalid argument
Sep 05 17:45:34 CamSVR bc-server[52775]: E(3/Driveway): failed to write snapshot frame
Sep 05 17:45:34 CamSVR bc-server[52775]: E(3/Driveway): Feeding snapshot decoder failed
Sep 05 17:45:34 CamSVR bc-server[52775]: E(3/Driveway): Failed to make snapshot
15

Okay, can you try 3.1.3? It should be in the repos now.

16

I tried repo and still shows 3.1.2. I did the install anyway. GUI shows 3.1.2.
Should I bet grabbing it somewhere else?

root@CamSVR:/tmp# apt list bluecherry
Listing... Done
bluecherry/bookworm 3:3.1.2 amd64 [upgradable from: 3:3.1.2]
N: There is 1 additional version. Please use the '-a' switch to see it
root@CamSVR:/tmp# apt list bluecherry -a
Listing... Done
bluecherry/bookworm 3:3.1.2 amd64 [upgradable from: 3:3.1.2]
bluecherry/now 3:3.1.2 amd64 [installed,upgradable to: 3:3.1.2]

root@CamSVR:/etc/apt/sources.list.d# cat bluecherry-bookworm.list
# Please post any problems to http://forums.bluecherrydvr.com or debian@bluecherrydvr.com
deb http://unstable.bluecherrydvr.com bookworm main # Bluecherry Debian 12 packages
17

Sorry, try one more time. Packages were uploaded to dl.bluecherrydvr.com not unstable

18

I updated the repo and upgraded. The system is now on 3.1.3 per GUI. After about 9 minutes of the system reboot memory usage was at 13G of 16G.

1 Like
19

now using 3.1.3 on 24lts for one server and 3.1.3 on 22lts for another server and i see no change in the memory problem. it installed fine though.

20

We are still trying to find the source for this memory leak. Currently the recommendation is downgrade to 3.1.1:

sudo apt install bluecherry=3:3.1.1

21

unfortunately that method of downgrading doesn’t work for ubuntu 24lts

1 Like
22

Hi @ironsmith, sorry for that. We are working to fix the memory leaks.

How bad is the memory leak, in KB/s? How long does the server process survive?

If your system has some CPU and RAM headroom, could you please run the Bluecherry server under a checker for a while? That would help us a lot to fix first what hurts you most.

  1. Make sure you’re running bluecherry 3.1.3: dpkg -l bluecherry | grep 3.1.3
  2. apt update && apt install bluecherry-dbg valgrind
  3. Edit /usr/lib/systemd/system/bluecherry.service to run under valgrind: comment out ExecStart line and put this one immediately after:

ExecStart=/usr/bin/valgrind --tool=memcheck --leak-check=yes --leak-check=full --show-leak-kinds=all --log-file=/var/log/bluecherry-valgrind.log /usr/sbin/bc-server -s

  1. Make the stage and restart Bluecherry server:
touch /var/log/bluecherry-valgrind.log
chown bluecherry /var/log/bluecherry-valgrind.log
systemctl daemon-reload
systemctl restart bluecherry

  1. Wait, perhaps an hour or so, or for as long as the server survives.

  2. Collect the logs:

systemctl stop bluecherry
cp /var/log/bluecherry-valgrind.log ~
gzip ~/bluecherry-valgrind.log
cp /var/log/bluecherry.log ~
gzip ~/bluecherry.log
  1. Switch to running without valgrind
systemctl daemon-reload
systemctl restart bluecherry
  1. Send the logs (bluecherry-valgrind.log.gz, bluecherry.log.gz) to us, for instance, email them to maintainers@bluecherrydvr.com

Thanks a lot.

23

I’m seeing a memory leak running 3.1.3 as well on Ubuntu 24.04.1 LTS:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                            
 290431 blueche+  20   0   17.6g  11.3g   9280 S 197.0  36.2     8d+9h bc-server 

$ dpkg -l bluecherry
ii  bluecherry     3:3.1.3      amd64        Bluecherry DVR Server

I’ve collected valgrind logs as instructed above and emailed them.

24

Can you try this version and let me know the results?

wget http://lizard.bluecherry.net/~curt/releases/3.1.4/jammy/bluecherry_3.1.4_amd64.deb
sudo dpkg -i bluecherry_3.1.4_amd64.deb
25

I’m on Noble (24.04.1 LTS), that sounds Jammy specific?

26

Try this release (noble)

27

seems better.

at start:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                            
 714198 blueche+  20   0 2571920 110404  41120 S 216.7   0.3   0:07.57 bc-server

after 5 hours:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                            
 714198 blueche+  20   0 2572052 153368  12480 S 197.0   0.5 303:41.26 bc-server
28

Fingers crossed.

Can you confirm accessing recordings, live streaming, etc all work for you?

29

The Debian12 server was upgraded to version 3.1.4 as per recommendations from bluecherry. I do not see the memory leak issue that was present on .2 and .3 .versions. Thanks for the fix. Cheers

30

Thanks for the report!