Artificial Intelligence & Machine Learning

ChatGPT is about to be turned into yet another propaganda tool by News Corp. just in time for the elections. This is a disgusting and disappointing move by Open AI. News Corp themselves have pleaded several times that no person in their right mind would believe they are actually news, but this is what they are going to use as journalism? Please voice your concern by unsubscribing from their service and finding another tool. Make sure you specify why you unsubscribed.

To cancel your ChatGPT Plus subscription, follow these steps:

1. Log into ChatGPT: Visit the ChatGPT website and log in with your credentials.

2. Access Subscription Settings: Click on "My Plan" in the left sidebar.

3. Manage Subscription: In the pop-up window, click "Manage my subscription."

4. Cancel Plan: On the checkout page, select "Cancel Plan." Your cancellation will take effect the day after your next billing date. To avoid being charged for the next billing period, ensure you cancel at least 24 hours before your next billing date.

For me its "Grandpadawan" XD

I asked my ChatGPT to pick a name that it liked for itself. It picked Echo. Does it say the same thing for everybody?

I regularly find myself saying things like "Can you ...." or "Do it again for this please". Are you polite, neutral, or rude to AI?

A few days ago, rgerganov's RPC code was merged into llama.cpp and the old MPI code has been removed. So llama.cpp supports working distributed inference now. You can run a model across more than 1 machine. It's a work in progress and has limitations. It currently is limited to FP16, no quant support yet. Also, I couldn't get it to work with Vulkan. But considering those limitations, it works pretty well. Inference is limited by network bandwidth. Using a 1 gigabit ethernet connection is faster than using a slower wifi connection. And the overall speed seems to be limited by the slowest machine. See my numbers below.

You can read more about it here.

Here are some numbers between a M1 Max Studio and a PC with a 7900xtx. The model is Tiny Llama FP16.

This first set of numbers is from the Mac as the client.

Mac only

llama_print_timings: prompt eval time =     199.23 ms /   508 tokens (    0.39 ms per token,  2549.77 tokens per second)
llama_print_timings:        eval time =    8423.24 ms /   511 runs   (   16.48 ms per token,    60.67 tokens per second)

7900xtx only

llama_print_timings: prompt eval time =     100.50 ms /   508 tokens (    0.20 ms per token,  5054.98 tokens per second)
llama_print_timings:        eval time =   10574.48 ms /   511 runs   (   20.69 ms per token,    48.32 tokens per second)

Mac + 7900xtx

llama_print_timings: prompt eval time =     230.29 ms /   508 tokens (    0.45 ms per token,  2205.92 tokens per second)
llama_print_timings:        eval time =   11147.19 ms /   511 runs   (   21.81 ms per token,    45.84 tokens per second)

Here are numbers from the 7900xtx PC as the client.

Mac only

llama_print_timings: prompt eval time =     253.78 ms /   508 tokens (    0.50 ms per token,  2001.77 tokens per second)
llama_print_timings:        eval time =   10627.55 ms /   511 runs   (   20.80 ms per token,    48.08 tokens per second)

7900xtx only

llama_print_timings: prompt eval time =      40.93 ms /   508 tokens (    0.08 ms per token, 12412.34 tokens per second)
llama_print_timings:        eval time =    4249.10 ms /   511 runs   (    8.32 ms per token,   120.26 tokens per second)

Mac + 7900xtx

llama_print_timings: prompt eval time =     198.44 ms /   508 tokens (    0.39 ms per token,  2559.98 tokens per second)
llama_print_timings:        eval time =   11117.95 ms /   511 runs   (   21.76 ms per token,    45.96 tokens per second)

As you can see, the inference overall seems to be limited by the speed of the network connection. Which is about 46t/s for this model. Even though both the Mac and the 7900xtx are faster than 48t/s locally, they are limited to 48t/s when run remotely.

To further illustrate that the network is the bottleneck, here's the numbers for the Mac running over wifi instead of ethernet.

llama_print_timings: prompt eval time =     737.93 ms /   508 tokens (    1.45 ms per token,   688.41 tokens per second)
llama_print_timings:        eval time =   42125.17 ms /   511 runs   (   82.44 ms per token,    12.13 tokens per second)

It's only 12t/s for TG versus 48t/s.

One last number for number sake. Here's the llama 3 7B model at FP16 running across both.

llama_print_timings: prompt eval time =     826.07 ms /   508 tokens (    1.63 ms per token,   614.96 tokens per second)
llama_print_timings:        eval time =   29902.27 ms /   511 runs   (   58.52 ms per token,    17.09 tokens per second)