Everyday around 1:00 pm I noticed that sudo apt dist-upgrade -s is running taking a lot of CPU. Enough that it maxes out the rest of the processor and ROS topics start to lag and timing gets off. My PID control loops are now operating on stale feedback topics and it isn’t pretty.
It isn’t clear to me what is causing the updates to kick off each day. I’ve disabled a handful of relevant systemd services. I’ve disabled a handful of cron jobs that seem relevant. I’ve attached a monitor and logged in through the gui and disabled software updates in system settings UI.
My machine is unusable for several minutes. I can typically kill -9 [pid] of the sudo apt dist-upgrade process and get it to stop but I’m gearing up for 24/7 autonomous operations and that isn’t really an option.
What am I missing? How do I disable daily updates?
Thanks,
Kyle
I bet others will chime in but I had a similar issue on a robot using Ubuntu 18.04. For the Ubuntu updates I found a way to turn off ubuntu level updates and this is my note. Perhaps search for these keywords for more info than I have here. Hope this helps.
On 18.04 automatic updates are set when unattended-upgrades is set to 1 in /etc/apt/apt.conf.d/20auto-upgrades
So I don’t have that system now but I believe I set that value to 0 and they stopped.
Again, somebody else may add more. Hope it helps.
Thanks for the reply. I failed to mention that I did verify that unattended updates were disabled.
As I continue to learn about this it seems that the -s flag means it isn’t actually installing any packages. It is just checking which packages need to be updated. This got me wondering if this is for the sake of updating the motd when logging in?
When I run sudo apt dist-upgrade -s manually it only takes a few seconds to complete. When testing on the machine it takes minutes. I have the pi connected to a hot spot and the signal is a little weak. I’m wondering if sudo apt dist-upgrade -s performs poorly and chews up CPU on unreliable internet connections. I’m not sure how I would verify it.
I’ve learned that I might be able to use ps auxf to determine what is calling it when it is running. I’m still very interested in getting this disabled.
The is really good feedback. I thought this was covered but apparently not. Thanks for bringing this up.
To help us more can you get me output from the uname -a please or better yet if you can tell me which image version you burned then we can duplicate the same situation.
Thanks
I’m using the 2019-06-19 image on a 3B+.
I’ve started testing with the new 2020-02-20 image in my dev environment on a Pi4 (4GB) but I haven’t ran it on one of our machines yet.
It also might be relevant that I’m running the 3B+ at about 92% memory utilization. I guess it is possible that apt dist-upgrade -s could run slower if available memory is limited. The lack of memory is why I’m working on switching to a Pi 4 with 4GB.
So 1pm ‘every day’ is that on the money or approximate.
Your statement on PID loops is curious. Are you running PID loops for wheel motor control on linux?
We are well aware of the Pi4 and are moving to that with more memory as well.
Approx. I typically notice it happening in the 1:00 pm hour. I’m wondering if it is randomized as part of a systemd timer / service.
Our general arch is all sensors connect to Arduino Mega. Rosserial connect Mega to Pi. RS485 connects Pi to AC VFDs. We are mostly using the ROS PIDController for our loops. Primarily an ackermann style steering system. The drive wheels are open loop controlled by VFDs.
This is a prototype with off the shelf hardware. We would not run PID on a non RT linux system in a production / industry environment.
Ok, thanks. I have passed on to ask a better Linux guy on the team to take a look at this post. I am mostly the hardware dude. Good luck. Mark
Hi @kylelanman,
Interesting that the update check is bogging down your system so much.
Can you try the following commands and see if they resolve your problem?
sudo systemctl disable apt-daily.service
sudo systemctl disable apt-daily.timer
sudo systemctl disable apt-daily-upgrade.timer
sudo systemctl disable apt-daily-upgrade.service
Please report back so that we can integrate these changes into future images.
Rohan
We have had enough other bugs the past couple days such that software updates have been the least of our worries. I’m hoping I can follow up late tomorrow or Thursday with my findings.
I did determine the root cause. We are leveraging the freedomrobotics.ai platform for monitoring and data logging. Using ps auxf while software updates were running I determined that their agent was kicking off the update check. I’ve reached out to them for further assistance.
I apologize for pointing the blame at Ubiquity initially. Thanks for being willing to dive in on this with me. This issue is resolved!
Thank you for the post indicating the issue is resolved.
Nice detective work on that one. These systems are complex at best.
Mark
Thanks for the update! Hopefully you get it sorted out with freedom robotics.
Rohan