Replace parquet.js with hyparquet

[platypii]

Uses hyparquet for javascript parquet parsing. It is a small, pure js implementation of parquet parsing with no dependencies. Parquet.js that this replaces is unmaintained and has not been updated in 5+ years.

Fixes #102 and #104 by using a well-maintained parquet library that supports modern parquet files.

I tested this with the parquet file generated by together-python and confirmed that upload works and fixes issue #104.

Let me know if I can help with anything!

---

Note

Switches parquet parsing to hyparquet and updates parquet file checks to use hyparquet metadata/schema APIs.

• Parquet handling:
  • Migrate _check_parquet in src/lib/check-file.ts to use hyparquet (asyncBufferFromFile, parquetMetadataAsync, parquetSchema) to read schema and row count.
  • Validate columns via children.map(...name) and sample count via metadata.num_rows; remove parquetjs reader logic.
• Dependencies:
  • Add hyparquet@1.14.0 and remove parquetjs and its type definitions from package.json.

^{Written by Cursor Bugbot for commit 6f1787b. This will update automatically on new commits. Configure here.}

[platypii]

@Nutlope? Anyone?

[nicolasembleton]

This should be straightforward to review.

[platypii]

@samselikoff?

[platypii]

Here's a video showing that instructions on together.ai website fail due to parquetjs parsing error. This PR fixes this issue:

together-upload.mp4

@Nutlope @samselikoff anything I can do to help move this along?

[Nutlope]

@platypii thanks so much for reporting and for the PR! We're in the process of fixing some things with the upload. @yogishbaliga, mind taking at this PR when you do your other PR on the upload functionality too?

[platypii]

@yogishbaliga thoughts? happy to contribute if there's more work that needs done

[blainekasten]

@platypii apologies for the delay on this. I'm actively maintaining these codebases now. I switched the base branch to next - could you address the conflicts and let me know when this is stable again. I would love to land this

[platypii]

@blainekasten thanks, I just rebased onto next branch, this should fix the Tokenized Data walkthrough on the together docs.

FYI there seems to be an issue with installing packages in this repo. When I run yarn install it calls the npm prepare script which includes a call to git-swap.sh which... deletes your entire local git repo and git history 😬 I THINK what you want is to move it from prepare to prepublish? I'm happy to put up another PR for that but it's not my repo so I didn't want to mess with packaging stuff. LMK.

Thanks!