Programmer Humor

21822 readers

2648 users here now

Welcome to Programmer Humor!

This is a place where you can post jokes, memes, humor, etc. related to programming!

For sharing awful code theres also Programming Horror.

Rules

Keep content in english
No advertisements
Posts must be related to programming or programmer topics

founded 2 years ago

MODERATORS

[email protected]

968

muskrat's data eng expert's hard drive overheats while processing 60k rows (lemmy.dbzer0.com)

submitted 1 week ago by [email protected] to c/[email protected]

297 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 366 points 1 week ago (64 children)

Wow.

I've been processing a couple of billion rows of data on my machine, the fans didn't even come on. WTF are they teaching "experts" these days, or has Elmo only hired people who claim that they can "wrangle data" and say "yes" ?

[–] [email protected] 5 points 1 week ago (1 children)

60k rows is generally very usable with even wide tables in row formats.

I’ve had pandas work with 1M plus rows with 100 columns in memory just fine.

After 1M rows move on to something better like Dask, polars, spark, or literally any DB.

The first thing I’d do with whatever data they’re running into issues with is rewrite it as partitioned and sorted parquet.

[–] [email protected] 4 points 1 week ago

My go-to tool of late is duckdb, comes with binaries for most platforms, works out of the box, loads any number of database formats and is FAST.

load more comments (62 replies)