Early in the history of docker, a lot of bits and bobs hadn’t been worked out yet, and I had a bug land on my desk where a service was leaking memory until it crashed, but only when running in a container. Turns out, the jvm at the time just never collected in a container because the /proc directory was mounted from the host rather than the k8s scheduler. So it would only collect if it did not receive a second allocation request during the GC.
This is no longer true with buildkit - you can use the --secret to securely pass a secret in as an argument.
I have found with individual dev environments that they cause many issues with outdated service versions. If you are going this route, I would use ScheduledScaler to shut down dev resources after hours.
I have found much more success with PR deployments - every PR gets deployed and wired up to a PR environment which has a full copy of dev and a copy of each PR which runs for 8 hours after the PR is built (I switch out the deploy for a job). Not every service maps well to this model, but it’s a good 95% solution in my experience.
This is an exit from a (iirc) 70mph underground highway to a 30mph surface road which seems to surprise people fairly often. But the exit has so many slow down signs and flashing lights, there is no excuse for the drivers.
On the other hand, this is clearly a danger to pedestrians, so something needs to be done - possibly closing the exit?
https://youtu.be/Q39uIXLh030