• 126 Posts
  • 3.61K Comments
Joined 1 year ago
cake
Cake day: September 1st, 2023

help-circle
  • Python has a Global Interpreter Lock (GIL) which has been a bane and a boon. A boon because many basic types are thread-safe as actions happen in lock step. A bane because despite having multiple threads, there’s still a master coordinating them all, which means there is no parallelism but concurrency. Python 3.13 allows disabling the GIL, but I cannot say much to that since I haven’t tested it myself. Most likely it means nothing is really thread safe anymore and it’s up to the developer to handle that.

    So, in Python, using multiple threads is not a surefire way to have a performance boost. Small tasks that don’t require many operations are OK for threading, but many cycles may be lost to the GIL. Using it for I/O bound stuff is good though as the main python thread won’t be stuck waiting on those things to complete (reading or writing files, network access, screen access, …) . Larger tasks with more operations that are I/O bound or require parallelism (encoding a video file, processing multiple large files at once, reading large amounts of data from the network, …) are better as separate processes.

    As an example: if you have one large file to read then split out into multiple small files, threads are a good option. Splitting happens sequentially, but writing to disk is (comparatively) slow task that one shouldn’t wait on and can be dedicated to a thread. Doing these operations on multiple large files is worth doing in parallel using multiple processes. Each process will read a file, split it, and write in threads, while one master process orchestrates the slave processes.

    Of course, your mileage may vary. I’ve run into the issue of requiring parallelism on small tasks and the only thing that worked was moving out that logic to a cython and outside the GIL (terrible experience). For small, highly parallel operations, probably Python isn’t the right language and something like Rust should be explored.

    Anti Commercial-AI license











  • Your post is nearly the epitome of Chesterton’s Fence. You don’t seem to understand why Rust looks the way it does, works the way it does, why it exists, what it’s used for, and what problems it solves, but you’re very happy (or not, which is probably why you wrote this post) to trash it.

    There are many responses to your comments that explain things quite well, yet, from what I see, you do not seem to concentrate on those.

    And what I quoted is just the icing on top. It looks very much like you have one style of programming and approaching problems (the PHP style of “if it runs, it’s good”) and apply it to every problem. You have used a hammer your whole life and every problem looks like a nail. You can build a good many things with duct tape, nails, and a hammer. It might all do the job well enough for your standards or purposes and at times it might even be the perfect tool for a task.
    But now you’ve discovered a screw driver, tried to hammer in a nail, and gotten quite frustrated that it didn’t work well. Instead of considering using a screw, you have tossed aside the screwdriver and decided to yell expletives into the ether.

    The ether has responded with explanations, but you have chosen to ignore them all and staunchly hold on to your “screwdrivers are shit” conclusion. Had you said “I’m just blowing off steam, don’t take this seriously”, that’s what it would’ve been. However, you seem quite serious. Or, as I said before, you’re just trolling.

    Anti Commercial-AI license