zky
/
ai_model
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '0555e21684'
${ noResults }
ai_model/runner
History
20918 5e0a8fedd0
Initial commit 		3 weeks ago
..
common Initial commit 3 weeks ago
llamarunner Initial commit 3 weeks ago
ollamarunner Initial commit 3 weeks ago
README.md Initial commit 3 weeks ago
runner.go Initial commit 3 weeks ago

README.md

runner

Note: this is a work in progress

A minimial runner for loading a model and running inference via a http web server.

./runner -model <model binary>

Completion

curl -X POST -H "Content-Type: application/json" -d '{"prompt": "hi"}' http://localhost:8080/completion

Embeddings

curl -X POST -H "Content-Type: application/json" -d '{"prompt": "turn me into an embedding"}' http://localhost:8080/embedding