What are the recommended scripting languages for complex shell scripts beyond bash?

Submitted ⁨⁨1⁩ ⁨year⁩ ago⁩ by ⁨CoderSupreme@programming.dev⁩ to ⁨programming@programming.dev⁩

I’ve been struggling with a rather complex shell script, and it’s becoming apparent that Bash might not be the best choice for this particular task. While I usually gravitate towards statically typed languages like Go or Rust, I’ve noticed that many people recommend alternative languages such as Lua or Python for scripting tasks.

I’m curious to know your opinions and experiences with scripting languages for larger or more intricate shell scripts. Have you ever encountered a situation where Bash just didn’t cut it, and if so, which scripting languages did you turn to for a more effective solution? Are there any specific languages you found particularly suitable for debugging, testing, or handling complex logic in your shell scripts?

source

Comments

Sort:hotnew top

MajorHavoc@lemmy.world ⁨1⁩ ⁨year⁩ ago
When I’m doing too much to maintain in bash, and not enough to merit Python, I use PowerShell.

My controversial take: if you’re looking for a better scripting language and haven’t tried PowerShell, you should give it a try.

It’s weird that Microsoft made a real shell.

PowerShell is actually open source, and it runs everywhere, including Mac and Linux. On Windows and Ubuntu, it’s already installed.

Powershell’s quality JSON and CSV handling is a huge game changer for quick scripts. The webrequest module is high quality. File operations are a breeze. Unlile bash, PowerShell can be formated to be pretty readable, when you care. Environment variable handling is mildly improved. Resusable code via modules is huge for quality of life.

PowerShell is the bash rewrite with lessons learned we all have wanted, but it’s not on a lot Linux folks radar because Microsoft published it.

source
- adept@programming.dev ⁨1⁩ ⁨year⁩ ago
  Alternatively try nushell. Its basically powershell without ms
  
  source
  - MajorHavoc@lemmy.world ⁨1⁩ ⁨year⁩ ago
    Oh, nice. I’ll check it out.
    
    source
- thesmokingman@programming.dev ⁨1⁩ ⁨year⁩ ago
  I’m trying to understand where you’d want to use PowerShell over Python. What’s something that’s in-between bash and Python?
  
  source
  - MajorHavoc@lemmy.world ⁨1⁩ ⁨year⁩ ago
    Good question.
    
    The Python subprocess module is a huge pain in the ass.
    
    PowerShell has about 80% of the power and readability of Python, while actually being a native shell.
    
    source
    -> View More Comments
- Thunderwolf@lemmy.world ⁨1⁩ ⁨year⁩ ago
  This 100%. I liked bash scripting when I was in college. Took some time to actually learn powershell, and it’s been amazing. Steep learning curve in the beginning, but it’s worth it
  
  source
- aluminium@lemmy.world ⁨1⁩ ⁨year⁩ ago
  not to mention that you have all the dotnet ecosystem at your fingertips. Wanna write a WPF application? Go ahead. You want to use ML.net? A bit clunky but doable.
  
  source
- ReluctantMuskrat@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Couldn’t agree more. It’s a great shell and scripting language. It’s object-oriented nature, native support for virtually every text format (csv, json, xml) and great libraries for others (yaml, excel), awesome regex and web/rest services support… it’s hard to beat and works on virtually every platform.
  
  Too few people in the Linux community will even look at it though since it has MS name on it.
  
  source
nous@programming.dev ⁨1⁩ ⁨year⁩ ago

for larger or more intricate shell scripts

Those are call applications. Use any language you like. If go/rust is what you know use them. I use rust all the time for things beyond run a bunch of commands and tends to be my go to when I need to process data in any way.

source
Nyfure@kbin.social ⁨1⁩ ⁨year⁩ ago
If the script gets too large or you think it will, write a full program.
With languages like python or go its not that hard either and you get alot of benifits aswell.

Other than that, use whatever languwge works and you know.

source
valen@lemmy.world ⁨1⁩ ⁨year⁩ ago
Unpopular opinion: I’m old school and would probably use perl.

source
- profoundlynerdy@programming.dev ⁨1⁩ ⁨year⁩ ago
  This is the correct answer. Perl is shell-like with support for advanced data structures and data parsing capabilities. Modern Perl is very slick, especially with the new object system.
  
  source
  - NostraDavid@programming.dev ⁨1⁩ ⁨year⁩ ago
    
    Modern Perl
    
    Perl or Raku?
    
    source
    -> View More Comments
- ook_the_librarian@lemmy.world ⁨1⁩ ⁨year⁩ ago
  I do most of my scripting in perl too. Python has always irked me.
  
  Only semi-related: I recently switched from bash shell to fish. I should have don’t that years ago. Don’t know why I held on to bash so tight.
  
  source
  - catastrophicblues@lemmy.ca ⁨1⁩ ⁨year⁩ ago
    What do you like about fish? Have you tried zsh? I’m in the market for a new shell too.
    
    source
    -> View More Comments
Knusper@feddit.de ⁨1⁩ ⁨year⁩ ago
Personally, I don’t feel like it’s worth learning a separate scripting language when you’re comfortable with a full-fledged programming language.

Python, Lua etc. used to be on a whole different level of usability, when compared to C. But compared to modern, high-level languages, the difference is marginal. Not to mention that at least compared to Rust, they start to look rather antique, too, and lack in robust tooling.

If you don’t feel like maintaining a whole git repo, use e.g. rust-script.

source
- beejjorgensen@lemmy.sdf.org ⁨1⁩ ⁨year⁩ ago
  A shell script can be more concise if you’re doing a lot of shell things. Keeps you from having os.system() all over the place.
  
  Things like “diff the output of two programs” are just more complex in other languages.
  
  I love rust, but replacing my shell scripts with rust is not something I would consider doing any more than I’d consider replacing rust with my shell scripts.
  
  source
  - Knusper@feddit.de ⁨1⁩ ⁨year⁩ ago
    Oh, I didn’t mean to say, you should throw out your shell scripts. For anything less than, say, 20 lines, they’re perfectly appropriate.
    
    I’m saying, Rust starts to feel like a good choice from, say, 100 lines upwards, and I just don’t think, it’s worth bridging the gap between those two.
    
    In particular, you can build a function that allows you to run commands without much boilerplate, e.g.: run(“echo hello | tee out.txt”);
    (The implementation just appends that argument to Command::new(“sh”).arg(“-c”) and runs it.)
    
    That way, you can do the more complex things in Rust, whether that’s control flow or something like modifying a JSON file, without giving up the utility of all the CLI tools on your system…
    
    source
    -> View More Comments
- philm@programming.dev ⁨1⁩ ⁨year⁩ ago
  Yeah as weird as it sounds to use a “low”-level systems programming language such as Rust. Rust works surprisingly well as “script” language. (And you don’t have to deal with the ugliness of bash, admittedly though, that bash is quite a bit more concise when using a lot of program executions and piping the results etc.)
  
  source
- pathief@lemmy.world ⁨1⁩ ⁨year⁩ ago
  I make bash scripts to automate the configuration of new servers. Stuff like install packages, create users, create groups, configure the database, manage permissions…
  
  I feel like that sort of stuff would be a nightmare to do in high level languages but maybe I’m just too used to bash.
  
  source
  - Knusper@feddit.de ⁨1⁩ ⁨year⁩ ago
    As I already responded to others, my comment was meant in the context of the question, so I would not learn a scripting language in addition to Bash + a programming language.
    
    For just running commands one-after-another, Bash is basically a minimal encoding, so no reason not to use it.
    
    When you do start to need if-elses, loops etc., that’s where Bash starts to become somewhat difficult to read. And personally, as someone who’s not fluent in Bash control flow, I found it quite useful to do the control flow in my programming language of choice, but still just calling commands like you’d do in Bash.
    
    Of course, this is a non-standard setup, and most target hosts will have Bash pre-installed, not rust-script, so it does obviously make a lot of sense to continue using Bash for what you’re doing.
    In general, my comment was meant for programmers. An ops person might know a full-fledged programming language, but still want to learn Python, because they need to write tons of Ansible tasks or whatever.
    
    source
ctr1@fl0w.cc ⁨1⁩ ⁨year⁩ ago
I usually use Awk to do the heavy lifting within my Bash scripts (e.g. arg parsing, filtering, stream transformation), or I’ll embed a Node.JS script for anything more advanced. In some cases, I’ll use eval to process generated bash syntax, or I’ll pipe into sh (which can be a good way to set up multiprocessing). I’ve also wanted to try zx, but I generally just stick to inlining since it saves a dependency.

source
a4ng3l@lemmy.world ⁨1⁩ ⁨year⁩ ago
My personal favourite remains perl for anything text oriented or for simple-ish orchestration.

source
rbos@lemmy.ca ⁨1⁩ ⁨year⁩ ago
I still reach for Perl when bash isn’t enough.

source
- tiredofsametab@kbin.social ⁨1⁩ ⁨year⁩ ago
  Same. I work primarily in Go and a little bit of Rust these days, but I still throw together a Perl script every year or two to automate something without needing to install something else on the machine or whatnot.
  
  source
starman@programming.dev ⁨1⁩ ⁨year⁩ ago
Try NuShell

source
ericjmorey@programming.dev ⁨1⁩ ⁨year⁩ ago
Look at Raku. This seems to be where Raku shines relatively brightly.

source
- steeznson@lemmy.world ⁨1⁩ ⁨year⁩ ago
  I think vanilla perl is also worth looking at if you need the script to be portable
  
  source
Heavybell@lemmy.world ⁨1⁩ ⁨year⁩ ago
Whatever works for you. I use C# for work so if there’s something I need to do that’s more complex than I know to do in bash I’ll just use that.

source
Lmaydev@programming.dev ⁨1⁩ ⁨year⁩ ago
I’ve used C# in the past. There’s a tool called dotnet-script that lets you run csharp files as scripts without having to compile.

It does require you to install dotnet though.

But tbh it’s pretty easy to just chuck an executable together.

source
brainw0rms@hexbear.net ⁨1⁩ ⁨year⁩ ago
I almost always use PowerShell (Core) for automation/scripting things that don’t warrant an entire “application”. It’s as powerful as you need it to be, but I wouldn’t recommend it if you aren’t already familiar with .NET and its ecosystem.

source
fartsparkles@sh.itjust.works ⁨1⁩ ⁨year⁩ ago
What are you trying to achieve in Bash that you’re struggling with? It’s hard to suggest alternatives without knowing your objectives since languages excel in different areas.

source
- CoderSupreme@programming.dev ⁨1⁩ ⁨year⁩ ago
  Temporary files management for my home folder, with archiving and cleanup after x and y days.
  
  source
  - oddityoverseer@lemmy.world ⁨1⁩ ⁨year⁩ ago
    You could look at logrotate if you don’t want to do something custom
    
    source
  - damium@programming.dev ⁨1⁩ ⁨year⁩ ago
    The expression syntax for the GNU find command is very powerful. I would expect that it is up to the task. If you don’t have the GNU find command with it’s extensions I could see how it’s would be difficult.
    
    source
  - toikpi@feddit.uk ⁨1⁩ ⁨year⁩ ago
    As @damium@programming.dev says you may be able to do this with find command. This command lists all PDF files under ~/tmp that were created more than 7 days ago and does a directory listing. You could use this as a basis to move create an archive of individual files.
    
    find ~/tmp -ctime +7 -iname “*pdf” -exec ls -rlht {} \;
    
    The find command also has a -delete flag.
    
    I have in the past used this combination to implement file management. I don’t have access to the script any more. I don’t remember why we used a shell script rather than logrotate as per @oddityoverseer@lemmy.world
    
    source
darcy@sh.itjust.works ⁨1⁩ ⁨year⁩ ago
i would say lua if possible, but python has more libraries

source
- UndercoverUlrikHD@programming.dev ⁨1⁩ ⁨year⁩ ago
  I’m curious on why you would recommend Lua if available. Is language speed a typical limiting factor for your scripts?
  
  I’m mostly asking since when I was forced to use the language it was nothing but annoyance. The 1 indexing and “if then/for do” will annoy me till my grave.
  
  source
  - darcy@sh.itjust.works ⁨1⁩ ⁨year⁩ ago
    just for simple tasks, where i need something between the simplicity of bash and the type safety of rust. its just my personal preference when it comes to scripting, although i do believe it should be the first consideration for choosing a user friendly but powerful embedded or config language, like for neovim, especially if performance is a concern, but it will ofc not always be the best option. the 1 index is certainly annoying though, but i would personally rather that than anything to do with python, especially whitespace. the if…then and for…do is the same as bash, so i dont think its that bad
    
    source
DirigibleProtein@aussie.zone ⁨1⁩ ⁨year⁩ ago
Python, Ruby

source
state_electrician@discuss.tchncs.de ⁨1⁩ ⁨year⁩ ago
I use Go a lot for this task, because it’s language I know well and can get the job done. And with Go you get a single binary, which is just as easy to deploy as a shell script.

source
Reptorian@programming.dev ⁨1⁩ ⁨year⁩ ago
For raster graphics image processing, I’d highly recommend G’MIC. Otherwise, Python.

source
WindowsEnjoyer@sh.itjust.works ⁨1⁩ ⁨year⁩ ago
Myself I know Bash and I know Go.

I would say it’s Python, so I use ChatGPT to code it for me if bash is not enough. If something that I can call “project” instead of “script” - I just use Go for it.

source
RoToRa@discuss.tchncs.de ⁨1⁩ ⁨year⁩ ago
Has anyone use Google’s zx and can say something about it?

source
- catastrophicblues@lemmy.ca ⁨1⁩ ⁨year⁩ ago
  zx looks super interesting! I’m building something very similar, except using Python as the language instead of JS: github.com/yrahul3910/pysh
  
  I’ll try out zx though, it looks cool, although I’d be worried about Google ditching it in the future.
  
  source
CMahaff@lemmy.world ⁨1⁩ ⁨year⁩ ago
I ran into this exact situation at work - though for me it was more the case that getting approvals for new software / installing new dependencies in our system is a massive pain.

So I went with Python since it’s already installed on basically any Linux system. It was fine - I mean Python is a good language and can certainly handle string processing and data manipulation with relative ease.

I still think the Python docs are pretty bad, and I wasn’t thrilled with the options for calling a subprocess in Python - they all felt kinda clunky, though I was barred from using the newest versions since I had to run an older version of Python.

But I ultimately got something that worked and it was certainly better executed / shorter than the bash equivalent it was replacing.

source
Kissaki@feddit.de ⁨1⁩ ⁨year⁩ ago
I’m mostly programming C# at my job (and private too now), and am quite fond of the language and ecosystem/basis. I tend to create even small utilities and one-off things with it. (Since C# 9 top level statements allow you to create the program entry point without a Main method - it becomes implicit/generated.)

I also find nushell quite interesting. It provides both a shell and scripting interface, and some promising concepts. I still have difficulties creating more complex command queries and scripts because the syntax is still foreign to me - and because you fall back to other tools when there’s a need.

source
snowe@programming.dev ⁨1⁩ ⁨year⁩ ago
Really surprised no one has mentioned Ruby. It’s installed by default on almost every system out there (unlike python), it will have the same features on every platform (unlike python where you might get 2.7 or 3.x depending. It’s simple and easy to read, and only slightly more verbose than bash. It’s very well suited for scripting (please don’t use it for application work). It also took a lot of its design from Perl, which a bunch of people are mentioning in this thread, and as a result has a ton of the features of perl, along with a ton of features from other languages. Rust is heavily based on Ruby’s design, and i’ve used Rust to create cli programs and I wouldn’t recommend it. It’s good, but most cli programs don’t need the difficulty of rust for the benefits that rust gives.

Anyway, python has a really really good cli library called Click, but that’s about the only good thing about it. If you are looking to use this script on multiple systems then Ruby will be much easier to transfer between systems (it will just work). I’ve deployed complex python, rust, and ruby CLIs across an org and Ruby was the only easy one. Rust was second easiest, Python absolutely terrible.

If you’re not deploying this to other platforms or sharing it across a team or something like that then a lot of the downsides and upsides here don’t really matter. Just use the easiest language.

source
- swordsmanluke@programming.dev ⁨1⁩ ⁨year⁩ ago
  Aw man, you can’t write all that and then not give an example!
  
  Ruby makes scripting drop-dead simple. You can run any shell command by surrounding it with back ticks.
  
  # simple example, just grab files: files = `ls`.split("\n") # pipes work inside back ticks files.map {|f| `cat #{f} | grep "can I use grep w/out cat"`} .compact .each { |match| puts match } # easy to build a pipeline on the data in ruby, too!
  
  That’s it! No messing around with popen3, or figuring out pipes or signals. Those are there too if you really need them, but if you just wanna write a quick script with a less arcane syntax - try Ruby!
  
  source
  - snowe@programming.dev ⁨1⁩ ⁨year⁩ ago
    Haha sorry, I wrote it all on my phone while traveling. and yeah, if you’re running just shell commands it looks almost the exact same as a bash script, and then when you need actual scripting capabilities you get them.
    
    source
- gopher_protocol@programming.dev ⁨1⁩ ⁨year⁩ ago
  Rust is heavily based on Ruby’s design
  
  I would not say “heavily based”. Literally only the closure/lambda syntax, which is cosmetic. Rust is mainly inspired by ML-family languages and C++.
  
  I think Ruby is a reasonable choice for small scripts which someone might otherwise use Python for. But Rust is very well suited to more complicated or long-lasting command-line tools, especially if performance is at all a concern. Clap alone is super nice, but there are a lot of awesome libraries for making rich CLI tools easily.
  
  colored or owo-colors for adding terminal colors super easily
  
  crossterm for cross-platform terminal feature support
  
  rustyline for all your fancy readline needs
  
  cursive for TUIs
  
  comfy-table for command-line tables
  
  And like…a hundred more I could mention. Idk, for anything that’s not completely trivial, which will be used and maintained by humans and not thrown away, Rust is really nice.
  
  source
  - snowe@programming.dev ⁨1⁩ ⁨year⁩ ago
    
    I would not say “heavily based”. Literally only the closure/lambda syntax, which is cosmetic. Rust is mainly inspired by ML-family languages and C++.
    
    All of Cargo is based on (and created by) the same person that created bundler for ruby. That list also misses out on a lot of things, like !, automatic returns, honestly most of the actual language ‘design’ rather than the internals (that seems to be a list of where the architects got their ideas for internal implementation as well, rather than just the readability of the language).
    
    But Rust is very well suited to more complicated or long-lasting command-line tools, especially if performance is at all a concern. Clap alone is super nice, but there are a lot of awesome libraries for making rich CLI tools easily.
    
    I disagree. Like I said, I wrote command line apps in all of these, performance was a factor. But a much larger factor is getting other devs on your team to contribute. And that was just absolutely impossible with Rust. The learning curve is just too high. For something that isn’t a hobby project, but that you might need a team member to roll out a fix in just a few hours, Rust will not cut it.
    
    Yes, you will have way more bugs in all the other programs, but honestly I had a shit ton of bugs in my rust cli as well, because, it turns out, rust works really well when it has control over everything, but man does it suffer when you have to interface with the real world… And oh boy did that make it incredibly difficult to write. Like I said, I deployed CLIs in all three of these languages. Ruby was the easiest of them all. Not just in development, but also maintenance.
    
    source
huginn@feddit.it ⁨1⁩ ⁨year⁩ ago
As an android developer working with GitHub actions I just use Kotlin Script: but that’s because I’m already writing in Kotlin all the time.

It’s really nice to be able to run it as a scripting language though

source
Longpork_afficianado@lemmy.nz ⁨1⁩ ⁨year⁩ ago
Maybe it’s becuase i grew up on python and bash still seems somewhat alien to me, but any time I’m crafting something with more than three or so nested functions, I use python as a wrapper for bash. Python initiates a bash script, parses the retval, inititates the next bash script from that data, etc.

source
- agressivelyPassive@feddit.de ⁨1⁩ ⁨year⁩ ago
  It’s really unfortunate, that the interaction between Bash and Python is so cumbersome. I would really like a simple Bash-class to invoke simple command strings directly.
  
  source
  - ms264556@beehaw.org ⁨1⁩ ⁨year⁩ ago
    It’s pretty close already. I forget where I cribbed the technique from, but I embed python functions into my scripts very often…
    
    E.g. here’s the relevant parts of a bash function from one of my hobby projects …
    
    function rks_encrypt { RUCKUS_SRC=“$1” RUCKUS_DEST=“$2” python3 - <
    
    source
catastrophicblues@lemmy.ca ⁨1⁩ ⁨year⁩ ago
I’m writing my own: github.com/yrahul3910/pysh

source
- darcy@sh.itjust.works ⁨1⁩ ⁨year⁩ ago
  nice
  
  source
AdrianContre@programming.dev ⁨1⁩ ⁨year⁩ ago
[deleted]
source
- AdrianContre@programming.dev ⁨1⁩ ⁨year⁩ ago
  [deleted]
  source
  - AdrianContre@programming.dev ⁨1⁩ ⁨year⁩ ago
    [deleted]
    source
    -> View More Comments
AdrianContre@programming.dev ⁨1⁩ ⁨year⁩ ago
[deleted]
source
- AdrianContre@programming.dev ⁨1⁩ ⁨year⁩ ago
  [deleted]
  source
  - AdrianContre@programming.dev ⁨1⁩ ⁨year⁩ ago
    [deleted]
    source
    -> View More Comments