Server 3.1.7 occasionally stops processing video from some cameras

Server discussion
1

I have noticed that occasionally by BC server 3.1.7 will stop recording the video from a camera. Other cameras on the same server continue to be recorded normally. The video that has stalled never recovers by itself and requires the bluecherry service to be restarted before it will start recording the video normally again.

The bluecherry.log file starts logging repeating messages for the affected camera like the following:

2024-11-18T12:29:02.871845+10:00 debian12-bluecherry bc-server[141179]: I(4/aid9701): Setting up device
2024-11-18T12:29:07.940092+10:00 debian12-bluecherry bc-server[141179]: I(4/aid9701): Stream started: h264, 1/90000(s) 1/90000(c); aac
2024-11-18T12:29:07.940308+10:00 debian12-bluecherry bc-server[141179]: I(4/aid9701): Switching to new recording schedule ‘continuous’
2024-11-18T12:29:07.974048+10:00 debian12-bluecherry bc-server[141179]: I(4/aid9701): Got bad dts=NOPTS on stream 0, passing to libavformat to handle
2024-11-18T12:29:13.451187+10:00 debian12-bluecherry bc-server[141179]: I(4/aid9701): Making a snapshot
2024-11-18T12:29:13.527897+10:00 debian12-bluecherry bc-server[141179]: I(4/aid9701): Saved snapshot from single keyframe
2024-11-18T12:32:38.366359+10:00 debian12-bluecherry bc-server[141179]: W(4/aid9701): Likely timestamping error. Ignoring.
2024-11-18T12:32:38.367985+10:00 debian12-bluecherry bc-server[141179]: W(4/aid9701): Likely timestamping(19352340) error. Ignoring.
2024-11-18T12:32:38.372411+10:00 debian12-bluecherry bc-server[141179]: I(4/aid9701): Got bad dts=19352340 while last was 19352430 on stream 0, setting to last+1
2024-11-18T12:32:38.372573+10:00 debian12-bluecherry bc-server[141179]: E(4/aid9701): Error writing frame to recording: Invalid argument
2024-11-18T12:32:38.372746+10:00 debian12-bluecherry bc-server[141179]: E(4/aid9701): Error writing frame to recording. Likely timestamping problem. Ignoring.

The corresponding BC server web client live view will show a image of what looks like perhaps the last successful image before the problem started, but the image never changes. I have attached a sample screen shot of the web client display when one of the recording is in this “stalled” state.

Live View
Live View755×428 122 KB

The windows v3 client will show a blank image where the recording should be with just a continuous loading message, and will not restart the video stream until after the server’s bluecherry service is restarted and the camera is closed and re-opened on the client.

The camera itself is working fine throughout the entire process as VLC and other tools show that the RTSP stream is working fine.

Can you offer any suggestions on how to debug this issue further and perhaps determine what might be triggering it in the first place.

1 Like
2

i have the same issue on two servers. it happens to me when one of the cameras goes down. upon coming back up bluecherry won’t properly connect with them again until the service is rebooted.

3

We believe we have identified the problem, we are working on resolving this.

4

Was this resolved as we are having a completely identical issue with version 3.1.7.

We were unable to roll-back to a previous version, but noticed checking for update a version 3.1.8 is available and got pushed down. Checking if that was a resolution.

5

Please try 3.1.8, we are working on releases notes but it should resolve the issue.

6

Did not fix the issue, in approximately a 24-hour period the cameras stop again.

7

Hi, I have same issue. On v 3.1.8. if camera is offline for while, bluecherry stop recording until I do server restart.

8

Hi @Martin, @Altherix , @ironsmith , @brett.gamlin ! Sorry that this is happening, and thanks for telling us.

Could you please tell, which schedule is Bluecherry server using for your camera? Is it “Continuous”, “Motion”, “Cont+Motion”, or a mix, so that it transitions from one mode to another? If you could send us substantial chunks of logs when this is happening, privately, that would be helpful.

@Martin when it is “offline for a while”, what duration of time was that?

9

We use continuous recording on all our servers.

The issue seemed to take approximately 14 hours than all recordings stopped and could not view any camera feeds.

In the log files, I didn’t see anything failing or odd but unfortunately didn’t save the logs.

I was unable to roll-back to a previous version, so had to uninstall BlueCherry and install 3.1.1 and do a backup restore of camera configuration.

It’s been over a week now with no issues on version 3.1.1, have stopped all BlueCherry updates on all servers until this issue is resolved.

10

@Altherix thanks for the details.
Would it be possible for you to set up a non-production Bluecherry 3.1.8 server in parallel, with at least one camera, to reproduce the issue and share the logs of that?

11

Hi, thank you, I also use just continuous recording with10 minuts long records. I have one camera. Duration of offline time can be about one minute. (It is duration of reboot camera.)

Here is error from log, but this error is not from the time of reboot camera. In time of reboot there is no log message.

After camera reboot, bluecherry server stop recording and live view in desktop app stop working, until I reboot bluecherry server.

bluecherry-error.log (2.9 KB)

12

I too am running continuous recording mode.

I do have a test system on which I reproduced the issue this morning by disconnecting one camera for a couple of minutes from 9:30am and 9:33am.

I checked /var/log/bluecherry.log but there is absolutely nothing recorded during the period of the camera being offline. I then rebooted the server at 9:35 to reset the recording. The log file shows:

2025-01-13T09:21:05.287894+10:00 debian12-bluecherry bc-server[612]: W(4/aid9701 test): Likely timestamping error. Ignoring.
2025-01-13T09:21:05.291144+10:00 debian12-bluecherry bc-server[612]: I(4/aid9701 test): Got bad dts=39158640 while last was 39158640 on stream 0, bailing out, causing the recording file to restart
2025-01-13T09:21:05.291365+10:00 debian12-bluecherry bc-server[612]: E(4/aid9701 test): Failure in recording writing
2025-01-13T09:23:15.061939+10:00 debian12-bluecherry bc-server[612]: I(4/aid9701 test): Got bad dts=4518976 while last was 4519672 on stream 1, bailing out, causing the recording file to restart
2025-01-13T09:23:15.062883+10:00 debian12-bluecherry bc-server[612]: E(4/aid9701 test): Failure in recording writing
2025-01-13T09:23:15.063036+10:00 debian12-bluecherry bc-server[612]: W(4/aid9701 test): Likely timestamping error. Ignoring.
2025-01-13T09:23:15.063163+10:00 debian12-bluecherry bc-server[612]: W(4/aid9701 test): Likely timestamping(4518976) error. Ignoring.
2025-01-13T09:23:15.330217+10:00 debian12-bluecherry bc-server[612]: W(4/aid9701 test): Likely timestamping error. Ignoring.
2025-01-13T09:23:15.330441+10:00 debian12-bluecherry bc-server[612]: W(4/aid9701 test): Likely timestamping(50849460) error. Ignoring.
2025-01-13T09:25:45.228165+10:00 debian12-bluecherry bc-server[612]: I(4/aid9701 test): Got bad dts=64336860 while last was 64336860 on stream 0, bailing out, causing the recording file to restart
2025-01-13T09:25:45.229170+10:00 debian12-bluecherry bc-server[612]: E(4/aid9701 test): Failure in recording writing
2025-01-13T09:25:45.229439+10:00 debian12-bluecherry bc-server[612]: W(4/aid9701 test): Likely timestamping error. Ignoring.
2025-01-13T09:35:08.548547+10:00 debian12-bluecherry bc-server[612]: I(): A new license request is accepted.
2025-01-13T09:35:08.588505+10:00 debian12-bluecherry bc-server[612]: I(): License is genuinely activated
2025-01-13T09:35:08.588883+10:00 debian12-bluecherry bc-server[612]: I(): A new license request is accepted.
2025-01-13T09:35:08.589021+10:00 debian12-bluecherry bc-server[612]: I(): Genuine Days left: unlimited

As for the timestamping errors that you see above, I get them quite often on this server and occasionally on two other 3.1.8 servers that I have running.

13

If you change the transport type from auto to UDP (in the Bluecherry → decide-> camera do you still have connection problems if you restart the camera?

14

Changing the transport to UDP did not resolve the issue of the server failing to reconnect to the camera’s rtsp stream after a camera goes offline for a short time. It still requires a restart of the bluecherry service or a reboot of the BC server before the server resumes recording the video feed.

Interestingly, even though the device settings for transport were changed to “UDP” in the bluecherry admin web page, netstat on the server showed that the server was still using TCP to connect to the cameras RTSP stream.

I tried this with both a HIKVision and a DaHua camera and they both exhibit the same symptoms.

15

No, change to UDP did not help.

16

Have you used 3.1.9? Is the problem resolved or does it continue to happen?

17

I haven’t seen this problem re-occur yet in any of my servers since I upgraded them to 3.1.9. The cameras streams seems to be working fine.

All I see in the logs now is occasional messages like:

2025-03-09T07:39:38.628003+10:00 mqn-vms-06 bc-server[840]: I(17/Cane Bin Workshop): Got bad dts=NOPTS on stream 0, passing to libavformat to handle
2025-03-09T07:39:38.786841+10:00 mqn-vms-06 bc-server[840]: I(17/Cane Bin Workshop): Bad timestamp: too large a gap of 1 seconds (1 tolerated), dts=7618961070 while last was 7617387060 on stream 0, bailing out, causing the recording file to restart
2025-03-09T07:39:38.787004+10:00 mqn-vms-06 bc-server[840]: E(17/Cane Bin Workshop): Failure in recording writing
2025-03-09T08:20:01.292559+10:00 mqn-vms-06 bc-server[840]: E(17/Cane Bin Workshop): Read error from stream: Connection timed out
2025-03-09T08:20:01.296083+10:00 mqn-vms-06 bc-server[840]: I(17/Cane Bin Workshop): Going to retry connection after a delay
2025-03-09T08:20:16.326361+10:00 mqn-vms-06 bc-server[840]: I(17/Cane Bin Workshop): Stream started: h264, 1/90000(s) 1/90000(c)
2025-03-09T08:20:16.345786+10:00 mqn-vms-06 bc-server[840]: I(17/Cane Bin Workshop): Back online
2025-03-09T08:20:16.356742+10:00 mqn-vms-06 bc-server[840]: I(17/Cane Bin Workshop): Got bad dts=NOPTS on stream 0, passing to libavformat to handle
2025-03-09T08:20:16.369055+10:00 mqn-vms-06 bc-server[840]: I(17/Cane Bin Workshop): Got bad dts=14940 while last was 7835166720 on stream 0, bailing out, causing the recording file to restart
2025-03-09T08:20:16.369108+10:00 mqn-vms-06 bc-server[840]: E(17/Cane Bin Workshop): Failure in recording writing
2025-03-09T11:35:10.832479+10:00 mqn-vms-06 bc-server[840]: I(17/Cane Bin Workshop): Got bad dts=NOPTS on stream 0, passing to libavformat to handle
2025-03-09T11:35:10.992577+10:00 mqn-vms-06 bc-server[840]: I(17/Cane Bin Workshop): Bad timestamp: too large a gap of 1 seconds (1 tolerated), dts=1053173340 while last was 1051484490 on stream 0, bailing out, causing the recording file to restart
2025-03-09T11:35:10.992832+10:00 mqn-vms-06 bc-server[840]: E(17/Cane Bin Workshop): Failure in recording writing

18

Thank you, update to 3.1.9. resolveed my issue too.

19

Thansa

Thanks, is this a wifi camera or a connection with intermittent internet?

20

That particular camera is a wired Dahua DH-IPC-HFW3866TP-ZAS-AUS 8MP bullet camera.

Although there have been some network outages at that site over the last week, none correspond to the times show in the bluecherry.log extract.

For the 9th Mar, the bluecherry server event log only shows a single alarm for that camera at 08:20am.