Many of these "local" Malay places are "nasi campur" places that are open for breakfast and lunch only and often close around 3pm.
"Nasi campur" has been described to me as the Malay version of "Nasi kandar" which is the Indian/Mamak version that tourists might be slightly more familiar with. I've also had the difference described to me as nasi kandar being just rice and curry while nasi campur has rice, curry, and other stuff. I'm still not sure which is technically correct but both definitions are helpful. I think translating directly is "mixed rice" for both. Westerners may think of these as "buffets" but if they associate the word "buffet" with all-you-can-eat then this will be less helpful as these places are "pay for what you take" rather thay "all you can eat". (Though you can often refill the rice.) Similar places with Chinese food are often but not always called "Economy rice".
The best such Malay places will have happy staff and happy customers and will have lots of customers. But it's sufficient if only the staff or only the customers are happy (-: If the staff seem baffled but other customers are friendly just ask them how it works.
Usually there's a disorganized line where at one end you take a plate and there'll be one or more large plastic containers with lids on that contain the rice and there will be a rice scoop inside. Some places will have multiple types of rice such as pandan rice. As well as plates there will be takeaway containers.
After you take your plate or container the line usually moves in both directions so just do your best to shuffle along from dish to dish in the trays to what looks yummy. To ask what certain things are I usually ask "in apa" - "what's this" but the answers usually only tell you the animal it came from. So you'll be told "ayam", "daging", "ikan" for "chicken", "beef", "fish". But you won't be told if it's regular flesh or organs or anything about the preparation. You might be told in English or Malay if it's spicy "pedas" but if it's a yellow or red curry/gravy/sauce it's probably spicy.
At Malay places you almost always just take your plate to a table and start to eat and a member of staff will come over, take a look, and give you a slip of paper with the amount. If you're not in a touristy area it's very likely to be lower than you expect. The same person or another person is likely to ask if you want a drink. The question is often "air?" pronounced "ire" which literally means "water" or sometimes the less ambiguous "minum" which means "drink". When you're finished you take the slip of paper with the price to the counter and pay. In some places it's hard to spot who to pay so in a new place I glance around trying to figure it out while I'm eating.
(At Chinese places that are similar you stop with your plate to show a person who tells you the price and you pay then before sitting.)
If you see the payment counter between selecting your food and sitting down you can probably pay first or they'll tell you in a friendly way to just take a seat. Malay places are very laid back and trusting so you can wander around looking for the hand-washing basin before paying. Often there are no tissues unless you ask.
Some mamak (Indian muslim) places are quite like this too. But some are much more like restaurants like you're used to with waiters milling around offering you a menu.
There are of course also Malay restaurants that are not "as local" as described but those are usually not confusing so don't need to be described here. They'll either have a menu at least on the wall or attached to the food cart. If it's a restaurant they'll treat you like any restaurant. If it's in a kopitiam or food court they'll typically only have two or three dishes at each stall, just as the Chinese, Indian, and Western stalls will.
