Can this be used for non-code semantic search?

[burntcouscous]

I've been trying this on non-code text files. E.g. searching for names or simply any topic related with default model (bge small) but unfortunately gives poor results (many ideal results not showing, and confidence scoring is not as you'd expect).

Could this be related bc I'm using bge small or is it using some other logic under the hood that makes it not work well for anything but code?

If likely bc of model, any documentation on how I use e..g openai embeddings to see if I can get better results?

[runonthespot]

definitely can work with regular non code - works on any text files by default (chunk + stride configurable), but also I added a rudimentary pdf adaptor.

What I'd say is these tiny embedding models aren't very intuitive in what they latch on to in search, so I created ck --tui so you can interactively explore the results and get a feel for what it might be useful for.

I'm definitely considering embedding models like openai - I use it a lot professionally, but I will need to think about cost & budget guardrails etc, BYOK. It's definitely on the list. I tried bigger local models but my ample MBP M2 Max still struggles. I'm especially tempted to try the mongo voyager embedding model which sounds like it might be a nice tradeoff of accuracy to size.

[burntcouscous]

The --tui switch is really cool (and great project and design in general).

But I'm afaid only these api based embeddings for now could do for the level of functionality I'm looking for. Is there any way to integrate a simple implementation as an 'alpha' obvious feature if yr worried abt first wanting guardrails etc? And not put it in the documentation e.g. .

e.g. ck --alpha-feat-openai-key="xxxx" --alpha-feat-api-embeddings-model="YYYY"

[runonthespot]

I'll add this next - I've been curious as to how much better something like an openai embedding model might perform for code search.

[burntcouscous]

Awesome.