How To Parse JSON Data To A Human Readable Format

Submitted ⁨⁨1⁩ ⁨day⁩ ago⁩ by ⁨irmadlad@lemmy.world⁩ to ⁨selfhosted@lemmy.world@lemmy.world⁩

So, yes this is selfhosting related. I am working on an n8n flow to pull in weather data so that I can have this data on a dashboard. I can’t find any dockerized weather forecasting apps. Most of them connect to a personal weather station, which might be an option in the future. For the time being, this is a little project I’m working on.

Partial JSON snippet:

spoiler

0 json cod “200” message 0 cnt 40 list 0 dt 1780693200 main temp 29.4 feels_like 29.23 temp_min 29.4 temp_max 29.68 pressure 1019 sea_level 1019 grnd_level 984 humidity 42 temp_kf -0.28 weather

I would like to display something like this:

spoiler

Current temperature: 23.25 °C Feels like: 24.09 °C Low / High: 23.25 °C / 23.60 °C Humidity: 94% Atmospheric pressure: 1023 hPa (sea level: 1023 hPa, ground level: 988 hPa) Temperature correction factor (temp_kf): -0.35 Weather: Light rain Weather code: 500 Short condition: Rain Icon: 10d

So, this is for you devs or coders out there. I can produce the JSON data. I’m just not sure how to parse it to something meaningful. I’m sure Python will have to be incorporated, but unsure of how to proceed.

Maybe someone could point me in a direction to tuts, articles, or your own experience. Sorry the JSON data doesn’t format correctly. Lemmy formatting doesn’t seem to allow that.

source

Comments

Sort:hotnew top

justme@lemmy.dbzer0.com ⁨6⁩ ⁨hours⁩ ago
isn’t json already a human readable format? ;) ok, jokes aside. to parse a file interactively, like i need to extract some information now, i use Julia, because I’m working with it anyway. that’s of course not very practical for automation scripts. most convenient would be a language that is already incorporated in your setup. otherwise i usually end up with a bash script with some jq calls.

source
K3can@lemmy.radio ⁨12⁩ ⁨hours⁩ ago
Personally, whenever I need to process anything text-based, I use perl.

Read the json into a hash, parse the values if desired, then plug the values into an html template.

It’s pretty quick to write, much easier to learn than python (in my opinion), and super powerful.

source
- irmadlad@lemmy.world ⁨9⁩ ⁨hours⁩ ago
  I know what perl is, but I have never used it before. I’ll spool up on that too.
  
  source
chrash0@lemmy.world ⁨23⁩ ⁨hours⁩ ago
i use Nushell for this! works with JSON, YAML, TOML, markdown, Polars Dataframes, SQLite, and a bunch of others including builtin parsing tools for whatever formats and a plugin ecosystem. i use it at work and for personal projects as my main shell, and it’s super handy for exploring, unpacking, sorting, and visualizing all sorts of data. i use it to:

find specific parts of YAML cloud configs

visualize JSON logs, including a parser that restructures journalctl logs.

_re_structure data from CLIs to work with them as structured: git logs, Unix coreutils, etc

script my environment: common kubectl queries, specific web API helpers, building and running and testing applications, etc

it is a slight learning curve, and technically you could do all of that with bash or zsh and jq or jc, but i appreciate the modern take on your base shell terminal env.

it’s replaced both Python and Bash for me.
source
- irmadlad@lemmy.world ⁨22⁩ ⁨hours⁩ ago
  
  Nushell
  
  www.nushell.sh? I’ll add it to this evening’s reading material. I was skimming the site and saw this: www.nushell.sh/cookbook/jq_v_nushell.html in relation to jq.
  
  Thanks for the input.
  
  source
  - chrash0@lemmy.world ⁨21⁩ ⁨hours⁩ ago
    exactly! i basically live in the terminal, and this is my go-to shell for all platforms
    
    source
GreenKnight23@lemmy.world ⁨23⁩ ⁨hours⁩ ago
lots of ways to do it, where will it be displayed? if it’s in a console, jq would more than suffice.

if you want it to be processed heavily and added to an HTML document, probably something like a mix of jq and envsubst. you could forego all that and just use sed to inject the json into HTML then have JavaScript parse it out for you browser side.

if you want to store it in a database why not just store the whole json?

your requirements aren’t very clear on what you want this data for or how it will be used. without that there’s just too many variable solutions.

can you parse json? sure! what for/how will it be used? not sure.

hope this helps.

source
- irmadlad@lemmy.world ⁨22⁩ ⁨hours⁩ ago
  
  envsubst sed
  
  Ahh more to explore. Thanks.
  
  your requirements aren’t very clear on what you want this data for or how it will be used.
  
  Apologies. As per usual, it’s all in my head and sometimes doesn’t make it to paper. I am using Homarr dashboard. Homarr has the ability to incorporate iframes. Ideally I would like to parse the JSON data into a prettier format perhaps adding some icons in the future once I get it all hammered out, and display it in an iframe on my dashboard.
  
  Thank you for the leads and your time.
  
  source
  - GreenKnight23@lemmy.world ⁨22⁩ ⁨hours⁩ ago
    I don’t know much about homarr but it sounds like going with jq and envsubst might give you a faster solution that you can just iframe in your dashboard. though the learning curve can be steep if you’re unfamiliar with Linux command line.
    
    I’m pretty sure you can literally pass in base64 encoded HTML to an iframe it will load in or pass it a publicly accessible file and it will work too.
    
    I had messed around with building a locally hosted dashboard using similar methods so I’m pretty sure it will work.
    
    source
yes@lemmy.dbzer0.com ⁨1⁩ ⁨day⁩ ago
You can point the homarr iframe to a HTML file that consumes the n8n json output with JavaScript (fetch and transform data)

Even easier might be to point the iframe at the n8n endpoint but to add a HTML node as last node to your workflow. There you can use plain HTML and it directly supports the expression syntax to render the values from your previous json node.

source
possiblylinux127@lemmy.zip ⁨1⁩ ⁨day⁩ ago
There are many different ways to do this

What is the usecase?

source
- irmadlad@lemmy.world ⁨1⁩ ⁨day⁩ ago
  Use Case: I want to display a daily and extended forecast on a dashboard perhaps via an iframe or other means. I get tired of going to the weather channel and having them complain about my ad blocking techniques. LOL Also, I thought it would be a good learning exercise for me. I’m using n8n to pull the raw data from OpenWeather API. I can generate the JSON data fine (at least it was an accomplishment for me), it’s just getting that JSON data into a prettier format to display on a dashboard with maybe some pretty little weather icons later on when I get it down pat.
  
  I mentioned Python because it seemed like a language I see others use for this sort of thing, but I am willing to explore other avenues. I am not very strong at coding. I can muck by, fiddling with this or that until I finally get something that works. Then I immediately commit that to my notes because my brain is shit at remembering things.
  
  source
  - obdurant@lemmy.zip ⁨1⁩ ⁨day⁩ ago
    I spent some time writing a script first in Python and then in ExcelScript at work to pull data from from OpenWeather API to compare ambient temperature and humidity to some of our indoor spaces. Both Python and JS (which ExcelScript is, basically) will make it very easy to pull the JSON from the API and pass whatever data you want down the line.
    
    I don’t have any experience with dashboards or your setup in particular so I can’t give any good advice about how to go forward though. If you have a pre-existing dashboard, what makes it and what feeds it data? Is it a webpage or an app or such?
    
    source
    -> View More Comments
  - possiblylinux127@lemmy.zip ⁨1⁩ ⁨day⁩ ago
    What dashboard are you using?
    
    source
    -> View More Comments
rimu@piefed.social@piefed.social ⁨21⁩ ⁨hours⁩ ago
In the Homarr docs, there is this: https://homarr.dev/docs/widgets/iframe/

Towards the end there is a link to https://github.com/diogovalentte/homarr-iframes which contains many examples of some content formatted nicely to fit within Homarr iframes and a docker container you can use to make a url that the iframe widget can consume. They used Go which is only a little harder than Python to read.

Seems like this one - https://github.com/diogovalentte/homarr-iframes/tree/main/src/sources/changedetectionio - makes a request to a JSON api, for example.

source
captcha_incorrect@lemmy.world ⁨22⁩ ⁨hours⁩ ago
Since you are using n8n, pull the data on a schedule and save it in a data table. Then create a webhook to serve the content as you want it. N8n can transform the JSON data, I can provide a sample tomorrow.

source
- irmadlad@lemmy.world ⁨8⁩ ⁨hours⁩ ago
  
  I can provide a sample tomorrow.
  
  That would be swell, but I wouldn’t make it a priority. Just whenever you get a round to it. I have been given a ton of source material to read. Stayed up a little too late last night trying to assimilate it all. LOL
  
  source
curbstickle@anarchist.nexus ⁨1⁩ ⁨day⁩ ago
You’ve already got some options for the json, so I’ll just add:

dockerized weather forecasting apps

As a thought, maybe magic mirror? I’m (ab)using magic mirror as a dakboard/skylight style calendar, no mirror to be found, and the weather widgets are pretty good too. Another option would be Home Assistant if you’ve already got it up and running, you can set kiosk style with a single page to a specific logged in user, same with grafana if you’ve got it up. The Infinity data source is great for json ingest.

source
- irmadlad@lemmy.world ⁨1⁩ ⁨day⁩ ago
  
  Home Assistant
  
  I thought about Home Assistant, but I’m not sure what I’d do with the rest of it’s capabilities. I live in a house built in the latter part of the 40’s sitting in the middle of 22 acres. Besides a monitor for all my security cams which I already have set up, there’s not a whole lot to automate with Home Assistant around here. LOL 2 rooms, one bath, part of which have the original logs from when it was a log cabin. Not a whole lot to keep up with.
  
  source
  - curbstickle@anarchist.nexus ⁨1⁩ ⁨day⁩ ago
    Oh see for me that sounds like a dream to set up HA for…. (also the kind of acreage I’m trying to get). Off the top:
    
    Solar, track the energy generation and battery storage/health
    
    Have outbuildings? Check on them with esphome sensors!
    
    Into astronomy? TRACK IT THERE!
    
    It mostly depends on what you’re doing with the property and what your goals might be. HA would be a lot to set up just for weather - magicmirror would definitely be the better choice for simple - but if you’ve got other stuff you are doing or want to do, HA would be a dream.
    
    source
    -> View More Comments
verw@lemmy.dbzer0.com ⁨1⁩ ⁨day⁩ ago
First of all, there might be more out-of-the-box tools out there for building dashboards that require little or no programming knowledge, but I’ve never used or looked for them. Microsoft’s Power BI is one that I’ve heard of in passing, but fuck micro$lop lol. It’s just the only example I can think of that might be more approachable. You could google something like, “open source power BI alternative” to find other options.

Here are a couple of tutorials about using Python to parse JSON:

w3schools tutorial for using Python to parse JSON

geeksforgeeks article explaining how to use Python to read JSON from a file

That will only cover how to interact with the JSON data, but not how to build a dashboard, ofc. Displaying the data in some fashion (e.g. a web page) is a whole other set of tasks.

If you’re new to programming in general, I recommend going through a collection of beginner tutorials on a programming language (Python is a good choice, but pretty much any will work for learning the basics). Any course like that will cover principles that apply to most languages and that would help you accomplish this task. For example, they’ll cover things like reading to and from files, parsing the content of files, and doing shit with the content.

Here are a few Python courses/tutorial collections you can check out. You don’t need to do all of them, but I wanted to give you some options so you’re more likely to find one that you vibe with:

w3schools Python tutorial

official Python tutorial

Python Basics: Introduction to Python - this is one of Real Python’s “learning paths” which are structured like a full-on course

I/O Operations and String Formatting - another course from Real Python. Goes more in depth on this particular topic and I would recommend this after learning the basics from one of the other resources I’ve mentioned.

EDIT: Btw, you are definitly not limited to using Python for this. Lot’s of other programming language will do. You just mentioned it, so I ran with that, haha.
source
- irmadlad@lemmy.world ⁨1⁩ ⁨day⁩ ago
  
  If you’re new to programming in general,
  
  I’m not new to programming, I’m just not very strong or proficient at all at it. You’ve given me a lot to read, and I thank you.
  
  Btw, you are definitly not limited to using Python for this
  
  I mentioned Python because that seems like a language a lot of people use for this sort of thing, but I am willing to entertain any other avenues…except perhaps Microsoft’s Power BI, which is discontinued.
  
  Thank you.
  
  source
  - curbstickle@anarchist.nexus ⁨1⁩ ⁨day⁩ ago
    Its not discontinued, btw, just now in the cloud and not free.
    
    Still would not recommend though, its awful imo.
    
    source