Simon:
> On 2024-03-21 15:22, karl@??? wrote:
> ...
> > A database management system like postgresql is just an unnessesary
> > overhead.
>
> This statement could create a false dichotomy in your mind.
>
> "A database management system" vs "text file"
>
> What about a "database" vs "text file"?
>
> Also somewhat misleading, since a text file could be considered a database.

What is it that you don't understand, any collection of data is a
database. If you then have some software acting like a server for
access that named data, then you have a database management system.

> If you do have your flat file in a common format such as CSV, well your
> making a database anyway - literally. You're just re-inventing the wheel.

No, I just make it easy for myself since you mostly use the latest set of
values.

> In my professional opinion, too many professionals are scared of the
> word "database". In my experience, they are really scared of relational
> databases, as they often jump on the "NoSQL" band wagon.
>
> Some "NoSQL" database might be a good fit for collecting large amounts
> of data, but if you know some SQL, you can start with SQLite (just a
> file) and then use a database server (MariaDB, PostgreSQL) later if you
> need to.

Yes, I know some sql, and I say it is a major overkill for data
collection and steering purposes.

But if you want to do some postprocessing, you might want to go that
way for the postprocessing part.

> A time series database sounds to be a good fit for your project.
> Prometheus is a decent one that I use. Though a time series database
> might not be necessary. https://en.wikipedia.org/wiki/Time_series_database

No, it isn't.

> The code to connect and write to and read data from a database is not
> complicated in most languages.

It is yet another connection which can fail.
Compare printf() to stdout to any db-code.

> ORMs are where you may find a lot of nonsense. With relational
> databases, the key (no pun) is to understand database normalization.
> https://en.wikipedia.org/wiki/Database_normalization

I just don't need normalization for a set of values like:
2017-05-05 23:59:58     52.0    48.2    10.3    34.7    32.0    20.3    35.2    23.9    8.9     10.3

> Now the real important reason why you want to use a database for your
> project:
>
> - There was that strange weather event some weeks ago and you want to
> query the data to see if sensor xyz also showed the anomaly.
> - Or you want to plot the difference between several sensors, motors,
> and load.
> - You want to see the average of xyz data (temp, fan speed, whatever).

gnuplot or a spreadsheet is just easy enough, or why not:
$ tail 2017-05-05.txt |
perl -e 'my $mean =0, $cnt =0; while(<>) { my @fld = split; $mean += $fld[9]; $cnt++ } print $mean/$cnt, "\n";'
23.9

Or you could fire up octave or R.

Regards,
/Karl Hammar