llama3.cuda
logits
- 프롬프트: I have a dream
[1, 76, 505, 263, 12561]5개 - 출력 (id와 logit값, 46개 출력으로 1개 더 많은 상태, CUDA)
[29889:13.076488] [940:12.449268] [12561:11.759573] [29879:15.145534] [310:13.017419] [263:8.233423] [4802:4.261223] [29892:4.124094] [9560:5.366050] [16423:5.078463] [2989:12.229407] [310:23.905329] [18281:7.510609] [322:14.829545] [10697:8.289629] [29889:15.419161] [940:14.181862] [12561:11.183266] [29879:17.269192] [310:10.015463] [8743:6.805388] [411:12.010337] [670:10.454911] [7875:9.869209] [322:14.330401] [321:9.460655] [1218:19.804911] [343:10.416383] [11770:22.203684] [5807:10.086064] [26514:23.825844] [29889:16.950533] [13:14.384112] [6716:19.083275] [2462:12.955539] [29892:11.870152] [540:11.442266] [471:12.093337] [22049:10.019634] [297:13.978985] [278:14.038999] [16423:13.947722] [746:13.010663] [540:15.065728] [4446:14.374954] [263:15.728765] - 출력 (Python)
[29889:13.105929362038037] [940:12.73067625436493] [12561:11.775138878217948] [29879:15.78920076722125] [310:12.843173873957994] [263:8.057945544400717] [4802:4.258677956772781] [29892:4.2125699986267735] [9560:5.341440519376919] [16423:5.154461198171746] [2989:11.990946787141965] [310:24.26916334946948] [18281:7.708171099718661] [322:14.909267287856556] [10697:8.18330034664377] [29889:15.33499487649984] [940:14.192116772451643] [12561:11.336398008635035] [29879:17.575418974756182] [310:10.190558156131136] [8743:6.872071118542993] [411:11.979582505929448] [670:10.62701438114209] [7875:9.677593240500208] [322:14.211040578581747] [321:9.454621240517747] [1218:20.024469182998658] [343:10.271546950227107] [11770:22.588724496973352] [5807:10.047320275561379] [26514:23.800264615276035] [29889:16.62579876333021] [13:14.47209195858064] [6716:19.150209322284365] [2462:12.972980729772905] [29892:11.863637377688187] [540:11.407205977456128] [471:12.026429451611941] [22049:9.843769819087358] [297:13.912187629375182] [278:13.992672100063404] [16423:14.024325453276948] [746:13.018670797002853] [540:15.09691628243275] [4446:14.45173913411657]
logit 값에 0.5% 정도 오차가 있다.
tokenizer.decode
디코드 결과 (CUDA)
I have a dream. He dreams of a big, beautiful garden full of flowers and trees. He dreams of playing with his friends and eating yummy snacks.
One day, he was walking in the garden when he saw
복수형으로 처리되는 차이가 있음.
디코드 결과 (Python)
I have a dream. He dream of a big, beautiful garden full of flower and tree. He dream of playing with hi friend and eating yummy snack.
One day, he wa walking in the garden when he saw
속도
- 4080 SUPER, cuBLAS, stories15M.bin: Token count: 50, elapsed: 0.017000s, 2823 tokens/s
- 4080 SUPER, naive matmul, stories15M.bin: Token count: 50, elapsed: 0.034000s, 1441 tokens/s
- 4080 SUPER, CPU(NumPy), stories15M.bin: 42 tokens/s
- 4080 SUPER, cuBLAS, stories110M.bin: Token count: 50, elapsed: 0.050000s, 1000 tokens/s
- 4080 SUPER, naive matmul, stories110M.bin: Token count: 50, elapsed: 0.433000s, 247 tokens/s
