Codesearch index limit hit

[taobojlen]

Codesearch has a hard-coded limit of 4GB for its index, and indexing the current RubyGems registry requires more than this. This means that codesearch isn't suitable for this project unless there's a workaround.

An alternative might be https://github.com/google/zoekt, but it has high memory requirements (the index today is ~90GB and it's all loaded into memory when searching).

[deivid-rodriguez]

So, is this not working at the moment? I don't want to start a command that might take days if it's not going to work 😅

[taobojlen]

Nope, it's not working -- Codesearch stops indexing partway through because of the limit.

You can still use this gem to download and extract all RubyGems packages, but you'll have to find your own solution for searching!

[deivid-rodriguez]

Thanks!

[hsbt]

I couldn't install zoekt on Debian 10 with golang provided by apt.

ubuntu@gem-codesearch:~$ go get github.com/google/zoekt
can't load package: package github.com/google/zoekt: found packages zoekt (api.go) and mirror (json.go) in /home/ubuntu/go/src/github.com/google/zoekt

[akr]

This reminds me #4
I'm not sure how it reduces the size.